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Abstract. Stochastic thermodynamics as reviewed here systematically provides 
a framework for extending the notions of classical thermodynamics like work, 
heat and entropy production to the level of individual trajectories of well-defined 
non-equilibrium ensembles. It applies whenever a non-equilibrium process is still 
coupled to one (or several) heat bath(s) of constant temperature. Paradigmatic 
systems are single colloidal particles in time-dependent laser traps, polymers 
in external flow, enzymes and molecular motors in single molecule assays, 
small biochemical networks and thermoelectric devices involving single electron 
transport. For such systems, a first-law like energy balance can be identified 
along fluctuating trajectories. For a basic Markovian dynamics implemented 
either on the continuum level with Langovin equations or on a discrete set 
of states as a master equation, thermodynamic consistency imposes a local- 
detailed balance constraint on noise and rates, respectively. Various integral 
and detailed fluctuation theorems, which arc derived here in a unifying approach 
from one master theorem, constrain the probability distributions for work, 
heat and entropy production depending on the nature of the system and 
the choice of non-equilibrium conditions. For non-equilibrium steady states, 
particularly strong results hold like a generalized fluctuation-dissipation theorem 
involving entropy production. Ramifications and applications of these concepts 
include optimal driving between specified states in finite time, the role of 
measurement-based feedback processes and the relation between dissipation and 
irreversibility. Efficiency and, in particular, efficiency at maximum power, 
can be discussed systematically beyond the linear response regime for two 
classes of molecular machines, isothermal ones like molecular motors, and heat 
engines like thermoelectric devices, using a common framework based on a cycle 
decomposition of entropy production. 



PACS numbers: 05. 40. -a: Fluctuation phenomena, random processes, noise, and 
Brownian motion; 05.70.Ln: Noncquilibrium and irreversible thermodynamics; 
82.37.-j: Single molecule kinetics; 87.16.Uv, Active transport processes 
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1. Introduction 

1.1. From classical to stochastic thermodynamics 

Classical thermodynamics, at its heart, deals with general laws governing the 
transformations of a system, in particular, those involving the exchange of heat, 
work and matter with an environment. As a central result, total entropy 
production is identified that in any such process can never decrease, leading, inter 
alia, to fundamental limits on the efficiency of heat engines and refrigerators. 
The thermodynamic characterization of systems in equilibrium got its microscopic 
justification from equilibrium statistical mechanics which states that for a system 
in contact with a heat bath the probability to find it in any specific microstate 
is given by the Boltzmann factor. For small deviations from equilibrium, linear 
response theory allows to express transport properties caused by small external fields 
through equilibrium correlation functions. On a more phenomenological level, linear 
irreversible thermodynamics provides a relation between such transport coefficients 
and entropy production in terms of forces and fluxes. Beyond this linear response 
regime, for a long time, no universal exact results were available. 

During the last 20 years fresh approaches have revealed general laws applicable 
to nonequilibrium system thus pushing the range of validity of exact thermodynamic 
statements beyond the realm of linear response deep into the genuine non-equilibrium 
region. These exact results, which become particularly relevant for small systems 
with appreciable (typically non-Gaussian) fluctuations, generically refer to distribution 
functions of thermodynamic quantities like exchanged heat, applied work or entropy 
production. 

First, for a thermostatted shear-driven fluid in contact with a heat bath, a 
remarkable symmetry of the probability distribution of entropy production in the 
steady state was discovered numerically and justified heuristically by Evans et al. [1]. 
Now known as the (steady state) fluctuation theorem (FT), it was flrst proven for 
a large class of systems using concepts from chaotic dynamics by Gallavotti and 
Cohen [2], later for driven Langevin dynamics by Kurchan [3] and for driven diffusive 
dynamics by Lebowitz and Spohn [4]. As a variant, a transient fluctuation theorem 
valid for relaxation towards the steady state was found by Evans and Searles [5] . 

Second, Jarzynski proved a remarkable relation which allows to express the free 
energy difference between two equilibrium states by a nonlinear average over the work 
required to drive the system in a non-equilibrium process from one state to the other 
[6, 7] . By comparing probability distributions for the work spent in the original process 
with the time-reversed one. Crooks found a "reflnemcnt" of the Jarzynski relation 
(JR), now called the Crooks fluctuation theorem [8, 9]. Both, this relation and another 
refinement of the JR, the Hummer-Szabo relation [10] became particularly useful for 
determining free energy differences and landscapes of biomolecules. These relations 
are the most prominent ones within a class of exact results (some of which found 
even earlier [11, 12] and then rediscovered) valid for nonequilibrium systems driven by 
time-dependent forces. A close analogy to the JR, which relates different equilibrium 
states, is the Hatano-Sasa relation that applies to transitions between two different 
non-equilibrium steady states [13]. 

Third, for driven Brownian motion, Sekimoto realized that two central concepts 
of classical thermodynamics, namely the exchanged heat and the applied work, can be 
meaningfully defined on the level of individual trajectories [14, 15]. These quantities 
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entering the first law become fluctuating ones giving birth to what he called stochastic 
energetics as described in his monograph [16]. Fourth, Maes emphasized that entropy 
production in the medium is related to that part of the stochastic action, which 
determines the weight of trajectories that is odd under time-reversal [17, 18]. 

Finally, building systematically on a concept briefly noticed previously [8, 19], 
a unifying perspective on these developments emerged by realizing that besides the 
fluctuations of the entropy production in the heat bath one should similarly assign 
a fluctuating, or "stochastic", entropy to the system proper [20]. Once this is 
done, the key quantities known from classical thermodynamics are defined along 
individual trajectories where they become accessible to experimental or numerical 
measurements. This approach of taking both energy conservation, i.e., the first law, 
and entropy production seriously on this mesoscopic level has been called stochastic 
thermodynamics [21], thus revitalizing a notion originally introduced by the Brussels 
school in the mid-eighties where it was used on the ensemble level for chemical 
nonequilibrium systems [22, 23]. 

1.2. Main features of stochastic thermodynamics 

Stochastic thermodynamics as here understood applies to (small) systems like colloidal 
particles, (bio)polymers (like DNA, RNA, and proteins), enzymes and molecular 
motors. All these systems are embedded in an aqueous solution. Three types of 
non-equilibrium situations can be distinguished for these systems. First, one could 
prepare the system in a non-equilibrium initial state and study the relaxation towards 
equilibrium. Second, genuine driving can be caused by the action of time-dependent 
external forces, fields, flows or unbalanced chemical reactions. Third, if the external 
driving is time-independent the system will reach a non-equilibrium steady state 
(NESS). For this latter class, particularly strong exact results exist. In all cases, even 
under such nonequililibrium conditions, the temperature of the system, which is the 
one of the embedding solution, remains well-defined. This property together with the 
related necessary time-scale separation between the observable, typically slow, degrees 
of freedom of the system and the unobservable fast ones made up by the thermal bath 
(and, in the case of biopolymers, by fast internal ones of the system) allows for a 
consistent thermodynamic description. 

The collection of the relevant slow degrees of freedom makes up the state of the 
system. Since this state changes either due to the driving or due to the ever present 
fluctuations, it leads to a trajectory of the system. Such trajectories belong to an 
ensemble which is fully characterized by the distribution of the initial state, by the 
properties of the thermal noise acting on the system and by specifying the (possibily 
time-dependent) external driving. The thermodynamic quantities defined along the 
trajectory like applied work and exchanged heat thus follow a distribution which can 
be measured experimentally or be determined in numerical simulations. 

Theoretically, the time-scale separation implies that the dynamics becomes 
Markovian, i.e., the future state of the system depends only on the present one with no 
memory of the past. If the states are made up by continuous variables (like position), 
the dynamics follows a Langcvin equation for an individual system and a Fokkcr- 
Planck equation for the whole ensemble. Sometimes it is more convenient to identify 
discrete states with transition rates governing the dynamics which, on the ensemble 
level, leads to a master equation. 

Within such a stochastic dynamics, the exact results quoted above for the 
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distribution functions of certain thermodynamic quantities follow universally for 
any system from rather unsophisticated mathematics. It is sufficient to invoke a 
"conjugate" dynamics, typically, but not exclusively, time-reversal, to derive these 
theorems in a few lines. Essentially, they lead to universal constraints on these 
distributions. One inevitable consequence of these theorems is the occurrence of 
trajectories with negative total entropy production. Such events have occasionally 
been called (transient) violations of the second law. In fairness to classical 
thermodynamics, however, one should emphasize that this classical theory ignores 
fluctuations. If the second law is understood as refering to the mean entropy 
production, it is indeed confirmed by these more recent exact relations. Moreover, 
they show that the probability for such events become typically exponentially small 
in the relevant system size which means that one has to sample exponentially many 
trajectories in order to observe these "violations" . 

Since these constraints on the distributions arc so universal, one might suspect 
that they are useless for uncovering system specific properties. Quite to the contrary, 
some of them offer a surprising relation between equilibrium and non-equilibrium 
properties with the JR as the most prominent and useful example. Moreover, such 
constraints can be used as an obvious check whether the assumptions of the model 
apply to any particular system. Finally, studying non-universal features of these 
distribution functions and trying to find further common aspects in these has become 
an important part of the activities in this field. 

Going beyond the thermodynamic framework, it turns out that many of the 
FTs hold formally true for any kind of Markovian stochastic dynamics. The 
thermodynamic interpretation of the involved quantities as heat and work is not 
mandatory to derive such a priori surprising relationships between functionals defined 
along dynamic trajectories. 

1.3. Hamiltonian, thermostatted and open quantum dynamics 

Even though I will focus in the main part of this review on systems described by 
a stochastic dynamics, it is appropriate to mention briefly alternative approaches as 
some of the FTs have originally been derived using a deterministic framework. 

Hamiltonian dynamics works, in principle, if the external driving is modelled 
by a time-dependent potential arising, e.g., from a movable piston, tip of an atomic 
force microscope, or optical tweezer. Conceptually, one typically requires thermalized 
initial conditions, then imagines to cut off the system from the heat bath leading to 
the deterministic motion and finally one has to reconnect the heat bath again. In a 
second variant, the heat bath is considered to be part of the system but one then has 
to follow all degrees of freedom. One disadvantage of Hamiltonian dynamics is that it 
cannot deal with a genuine NESS, which is driven by a time-indepedent external field 
or flow, since such a setting inevitably heats up the system. 

Thermostatted dynamics can deal with NESSs. Here, one keeps deterministic 
equations of motion and introduces a friction term making sure that on average the 
relevant energy (kinetic or total, depending on the scheme) does not change [24]. 

Even though a deterministic dynamics is sometimes considered to be more 
fundamental than a stochastic one, the latter has at least three advantanges from 
the perspective held in this review. First, from a practical point of view, in soft 
matter and biophysics a description focussing on the relevant (and measurable) degrees 
of freedom and ignoring water molecules from the outset has a certain economical 
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appeal. Second, stochastic dynamics can describe transitions between discrete states 
like in (bio)chemical reactions with essentially the same conceptual framework used 
for systems with continuous degrees of freedom. Third, the mathematics required for 
deriving the exact relations and for stating their range of validity is surpringly simple 
compared to what is required for dealing with NESSs in the deterministic setting. 

Open quantum systems will not be discussed explicitly in this review. Some of 
the FTs can indeed be formulated for these systems, sometimes at the cost of requiring 
somewhat unrealistic measurements at the beginning and end of a process, as reviewed 
in [25, 26]. The results derived and discussed in the following, however, are directly 
applicable to open quantum systems whenever coherences, i.e. the role of non-diagonal 
elements in the density matrix, can be ignored. The dynamics of the driven or open 
quantum system is then equivalent to a classical stochastic one. For the validity of the 
exact relations in these cases, the quantum-mechanical origin of the transition rates 
is inconsequential. 

1.4. Scope and organization of this review 

By writing this review, apart from focussing entirely on systems governed by 
Markovian stochastic dynamics, I have been guided by the following principles 
concerning format and content. 

First, I have tried to present the field in a systematic order (and notation) rather 
than to follow the historical development which has been briefly alluded to in the 
introductory section above. Such an approach leads to a more concise and coherent 
presentation. Moreover, I have tried to keep most of the more technical parts (some 
of which are original) still self-contained. Both features should help those using this 
material as a basis for courses which I have given several times at the University of 
Stuttgart and at summer schools in Beijing, Boulder and Jiilich. 

Second, as a consequence of the more systematic presentation, experimental, 
analytical and numerical case studies of specific systems are mostly grouped together 
and typically placed after the general theory where they fit best. 

Third, for the exact results the notions "theorem" , "equality" and "relation" are 
used here in no particular hierarchy. I rather try to follow the practice established in 
the field so far. In particular, it is not implied that a result called here "theorem" is 
in any sense deeper that another one called "relation" . 

This review starts in Sect. 2 by introducing a paradigm for this field which is a 
colloidal particle driven by a time-dependent force as it has been realized in several 
experiments. Using this system, the main concepts of stochastic thermodynamics, like 
work, heat and entropy changes along individual trajectories, will be introduced. At 
the end of this section, simple generalizations of driven one-dimensional motion such 
as three-dimensional motion, coupled degrees of freedom and motion in external flow 
are discussed. 

A general classification and a physical discussion of the major fluctuation 
theorems dealing with work and the various contributions to entropy production 
follows in Sect. 3. In Sect. 4, I present a unifying perspective on basically all known 
FTs for stochastic dynamics using the concept of a conjugate dynamics. It is shown 
explicitly, how these FTs follow from one master theorem. Sect. 5 contains an overview 
of experimental, analytical or numerical studies of Langevin-type dynamics in specific 
systems. Sect. 6 deals with Markovian dynamics on a discrete set of states for which 
FTs hold even without assuming a thermodynamic structure. 
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The second part of the review deals with ramifications, consequences and 
appHcations of these concepts. In Sect. 7, the optimal driving of such processes 
is discussed and the relation between time's arrow and the amount of dissipation 
derived. Both concepts can then be used to discuss the role of measurements and 
(optimal) feedback in these systems. Sect. 8 deals with generalizations of the well- 
known fluctuation-dissipation theorem to NESSs where it is shown that stochastic 
entropy plays a crucial role. 

Biomolecular systems are discussed in Sect. 9 where special emphasis is given to 
the role of time-scale separation between the fast (unobservable) degrees of freedom 
making up a well-defined heat bath for the non-equilibrium processes of the slow 
variables caused by mechanical or chemical imbalances. From a conceptual point of 
view the second essential new aspect of these systems is that each of the states is 
composed of many microstates which leads to the crucial notion of intrinsic entropy 
that enters some of the exact relations in a non-trivial way. 

Coming back to the issues that stood at the origin of thermodynamics, the final 
two sections discuss the efficiency and optimization of nano and micro engines and 
devices where it is useful to distinguish isothermal engines like molecular motors 
discussed in Sect. 10 from heat engines like thermoelectric devices treated in Sect. 11. 
A brief summary and a few perspectives are sketched in Sect. 12. 

1.5. Complementary reviews 

A selection of further reviews dealing with the topics discussed in the first part of this 
article can roughly be grouped as follows. | 

The influential essay [27] had an introductory character. More recent non- 
technical accounts have been given by Jarzynski [28] and van den Broeck [29] who 
both emphasize the relation of the fluctuation theorems with irreversibility and time's 
arrow. Other brief reviews by some of the main proponents include [17, 30, 31, 32] and 
the contributions in the collection [33] . Ritort has written a review on the role of non- 
equilibrium fluctuations in small systems with special emphasis on the applications 
to biomolecular systems [34] . A review focussing on experimental work by one of the 
main groups working on fluctuation theorems is [35]. 

For stochastic dynamics based on the master equation, a comprehensive derivation 
of FT's has been given by Harris and Schiitz [36]. The fluctuation theorem in the 
context of thcrmostatted dynamics has been systematically reviewed by Evans and 
Searles [37]. From the perspective of chaotic dynamics it is treated in Gallavotti's 
monograph [38]. The links between different approaches and rigorous mathematical 
statements are surveyed in [39, 40, 41]. 

Stochastic thermodynamics focusses on a description of individual trajectories 
as does an alternative approach by Attard introducing a "second entropy" [42]. 
On a more coarse-grained level, phenomenological thermodynamic theories of non- 
equilibrium systems have been developed inter alia under the label of "extended 
irreversible thermodynamics" [43], "GENERIC" [44], "mesoscopic dynamics of 
thermodynamic systems" [45] and "steady state thermodynamics" [46, 47]. 

Nice reviews covering related recent topics in non-equilibrium physics are [48, 49]. 

I Relevant reviews for the more specific topics treated in the second part of this review will be 
mentioned in the respective sections. 
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2. Colloidal particle as paradigm 

The main concepts of stochastic thermodynamics can be introduced using as a simple 
model system a colloidal particle confined to one spatial dimension, which can arguably 
serve as the paradigm in the field. 

2.1. Stochastic dynamics 

The ovcrdamped motion x{t) of a colloidal particle (or any other system with a 
single continuous degree of freedom) can be described using three equivalent but 
complementary descriptions of stochastic dynamics, the Langevin equation, the path 
integral, and the Fokker-Planck equation. 
The Langevin equation reads 

X = ^lF{x, A) + C = ^^{~^,V{x, A) + fix, A)) + C- (1) 

The systematic force F{x, A) can arise from a conservative potential V{x, A) and/or be 
applied to the particle directly as /(x, A). Distinguishing these two sources is relevant 
only in the case of periodic boundary conditions on a ring of finite length. Both 
sources may be time-dependent through an external control parameter A(t) varied 
from A(0) = Ao to \{t) = At according to some prescribed protocol. 
The thermal noise has correlations 

(C(r)C(r'))=2i?5(r-r') (2) 

where D is the diffusion constant. In equilibrium, D and the mobility /i are related 
by the Einstein relation 

D = Tn (3) 

where T is the temperature of the surrounding medium with Boltzmann's constant 
ks set to unity throughout this review to make entropy dimensionless. In stochastic 
thermodynamics, one assumes that the strength of the noise is not affected by the 
presence of a time-dependent force. The range of validity of this crucial assumption 
can be tested experimentally or in simulations by comparing with theoretical results 
derived on the basis of this assumption. 

The Langevin dynamics generates trajectories x{t) starting at x(0) = xq with a 
weight 

p[x(t)|xo] - AAcxp[-^([x(T), A(t)])] (4) 

where ^ 

A{[x{t),X{t)])= f dT[{x-iiFf/W + iid^,F/2] (5) 
Jo 

is the "action" associated with the trajectory. The last term arises from the 
Stratonovich convention for the discretization in the Jacobian when the weight for 
a noise history [C(t)] is expressed by [a:(r)]. This symmetric discretization is used 
implicitly throughout this review. Path dependent observables f2[a;(r)] can then 
be averaged using this weight in a path integral which requires a path-independent 
normalization M such that summing the weight (4,5) over all paths is 1. Throughout 
the review averages using this weight and a given initial distribution po{x) will be 
denoted by (....) as in 



{n[x{T)]) = dxo d[x{T)]n[x{T)]p[x{T)\xo]po{xo)- (6) 




Figure 1. Colloidal particle driven along a periodic potential V{x,\) by a non- 
conservative force /(A). In a NESS, the external parameter A is independent of 
time. 



Equivalently, the Fokker-Planck equation for the probability p{x, t) to find the 
particle at x at time r is 

drp{x,T) ^ - d^j{x,T) 

^ - {fiF{x, X)p{x, t) - DdM^, r)) (7) 

where j{x,T) is the probability current. This partial differential equation must 
be augmented by a normalized initial distribution p{x,0) = Po{x). For further 
calculations, it is useful to define a mean local velocity 

t) = t)/p{x, t). (8) 

More technical background concerning these three equivalent descriptions of 
Markovian stochastics dynamics of a continuous degree of freedom is provided in the 
monographs [50, 51, 52, 53]. 



2.2. Non- equilibrium steady states (NESSs) 

For time-independent control parameter A, any initial distribution will finally reach a 
stationary state p''{x, A). For / = 0, this stationary state is the thermal equilibrium, 

p-^ix, A) = exp[-(y(x. A) - J-(A))/T], (9) 

with the free energy 

TiX) = -Tin J dx cxp[^Vix,\)/T]. (10) 

A non-conservative force acting on a ring as shown in Fig. 1 generates a paradigm 
for a genuine non-equilibrium steady state (NESS) with a stationary distribution 

p^(x,A) =exp[-(/)(x,A)], (11) 

where ^(a;. A) is the "non-equilibrium" potential. In one dimension, p''{x,X) can be 
obtained explicitly by quadratures [51] or by an intriguing mapping to an equilibrium 
problem [54]. Characteristic for such a NESS is a steady current 

f = tiF{x)p'[x) - Dd^p'ix) EE v%x)p''{x) (12) 
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with the mean local velocity (x) . Even for time-dependent driving, one can express 
the total force 

F{x, A) = [v'{x, X) - Dd^c^ix, X)]/ii (13) 

through quantities refering to the corresponding stationary state which is sometimes 
helpful. 

Occasionally, we will use (...}°'^ and (...)* to emphasize when averages or 
correlation functions are taken in genuine equilibrium and in a NESS, respectively. 



2.3. Stochastic energetics 

2.3.1. The first law Sekimoto suggested to endow the Langevin dynamics with a 
thermodynamic interpretation by applying the notions appearing in the first law 

dw = dE +dq (14) 

to an individual fluctuating trajectory [14, 15]. Throughout the article, we use the 
convention that work applied to the particle (or more generally system) is positive as 
is heat transferred or dissipated into the environment. 

It is instructive first to identify the first law for a particle in equilibrium, 
i.e. for / = and constant A. In this case, no work is applied to the system 
and hence an increase in internal energy, defined by the position in the potential, 
dE = dV ~ {dxV)dx = —dq, must be associated with heat taken up from the reservoir. 

Applying work to the particle either requires a time-dependent potential 
V{x,X{t)) and (or) an external force /(x, A(t)). The increment in work applied to 
the particle then reads 

dw = (dV/dX) dX + f dx, (15) 

where the first term arises from changing the potential at fixed particle position. 
Consequently, the heat dissipated into the medium must be identified with 

dq^dw-dV = Fdx. (16) 

This relation makes physical sense since in an overdamped system the total force 
times the displacement corresponds to dissipation. Integrated over a time interval t, 
one obtains the expressions 

w[x{t)]= f [{dV/dX)X + fx]dT (17) 
Jo 

and ^ ^ 

q[x{T)]= / drq^ Fxdr (18) 
Jo Jo 

and the integrated first law 

w[x{t)] ^ q[x{T)] +AV^ q[x{T)] + V{xt,Xt) ~ V{xo,Xo) (19) 

on the level of an individual trajectory. 

The expression for the heat requires a prescription of how to evaluate Fx. As 
above in the path integral, one has to use the mid-point , i.e., Stratonovitch rule for 
which the ordinary rules of calculus for differentials and integrals apply. 
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The expression for the heat dissipated afong the trajectory x{t) can also be 
written in the form 

q[x[T)\ = T — — rrr = T\n 20 

A{[x{t-T),\{t-T)]) p[x{t),X{t)] 

as a ratio involving the weight (5) for this trajectory given its initial point xq compared 
to the weight of the time- reversed trajectory x{t) = x{t — t) under the reversed 
protocol A(t) = X(t — t) for xq = x(t) = Xf. This formulation points to the deep 
relation between dissipation and time reversal which repeatedly shows up in this field. 



2.3.2. Housekeeping and "excess" heat Motivated by steady state thermodynamics, 
it will be convenient to split the dissipated heat into two contributions [46, 13] 



Q^g'^'^ + g^^ (21) 



The housekeeping heat is the heat inevitably dissipated in maintaining the 
corresponding NESS. For a Langevin dynamics, it reads 

ql^k^ f dTx{T)^l-H'{x(T),\(T)). (22) 

Jo 

The "excess" heat 

g°''= -{D/^i) f dTx{T)d,^ix,X) 
Jo 

= T[-A(t)+ [ dT\dx(l)] (23) 

"'0 

is the heat associated with changing the external control parameter where we have 
used (13,18). 



2.3.3. Heat and strong coupling This interpretation of the first law and, in particular, 
of heat relies on the implicit assumption that the unavoidable coupling between 
particle (or, more generally, system) described by the slow variable x and the degrees 
of freedom making up the heat bath does neither depend crucially on x nor on the 
control parameter A. Such an idealization may well be obeyed for a colloidal particle 
in a laser trap but will certainly fail for more complex systems like biomolecules. In 
the following, we first continue with this simple assumption. In Sect. 9.2, we discuss 
the general case and point out which of the results derived in the following will require 
a ramification. Roughly speaking, most of the fluctuation theorems hold true with 
minor modifications whereas infering heat correctly indeed requires one more term 
compared to (16). 



2.3.4- Alternative identification of work The definition of work (15) has been 
criticised for supposedly being in confiict with a more conventional view that work 
should be given by force times displacement, see [55] and, for rebuttals, [56, 57, 58]. 
In principle, such a view could be integrated into the present scheme by splitting the 
potential into two contributions. 



F(x,A) = F°(x-,Ao) + y'='^(x,A), 



(24) 
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the first being an intrinsic time-independent potential, and the second one a time- 
dependent external potential used to transmit the external force. If one defines work 
as 

dw'''' = {-d^V'^ix, A) + f)dx, (25) 
it is trivial to check that the first law then holds in the form 

dw'''' = dE"'' +dq (26) 

with the corresponding change in internal energy dE°^ = dV'^ = d^V^dx and the 
identification of heat (16) unchanged. Clearly, within such a framework, it would be 
appropriate to identify the internal energy with changes in the intrinsic potential only. 
Integrated over a trajectory, this definition of work differs from the previous one by a 
boundary term. Aw = Aw°^ + AV°^. 

It is crucial to appreciate that exchanged heat as a physical concept is, and should 
be, independent of the convention how it is split into work and changes in internal 
energy. The latter freedom is inconsequential as long as one stays within one scheme. 
A clear disadvantage of this alternative scheme, however, is that changes in the free 
energy of a system are no longer given by the quasistatic work relating two states. 
In this review, we will keep the definitions as introduced in Sect. 2.3.1 and only 
occasionally quote results for the alternative expression for work introduced in this 
section. 



2.4- Stochastic entropy 

Having expressed the first law along an individual trajectory, it seems natural to 
ask whether entropy can be identified on this level as well. For a simple colloidal 
particle, the corresponding quantity turns out to have two contributions. First, the 
heat dissipated into the environment should obviously be identified with an increase 
in entropy of the medium 

As'^^ixir)] ^ q[xiT)]/T. (27) 

Second, one identifies as a stochastic or trajectory dependent entropy of the system 
the quantity [20] 

s{t) = -In p{x{t),t) (28) 

where the probability p{x, r) obtained by first solving the Fokker-Planck equation is 
evaluated along the stochastic trajectory x{t). Thus, the stochastic entropy depends 
not only on the individual trajectory but also on the ensemble. If the same trajectory 
x(t) is taken from an ensemble generated by another initial condition p(x,0), it will 
lead to a different value for s(r). 

In equilibrium, i.e., for / = and constant A, the stochastic entropy s(r) just 
defined obeys the well-known thermodynamic relation, TS = E — F, between entropy, 
internal energy and free energy in the form 

Ts{t)^V{x{t),X)-T{X), (29) 

now along the fluctuating trajectory at any time with the free energy defined in (10) 
above. 

Using the Fokker-Planck equation the rate of change of the entropy of the system 
(28) follows as [20] 



drp{x,T) 

s(r) =- 



p{x, t) 



( j{x,T) _ fiFjx, A) 



x{T] \Dp{x,t) D y'^(^)^' ^^^^ 
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Since the very last term can be related to the rate of heat dissipation in the medium 
(18), using D = T^, one obtains a balance equation for the trajectory-dependent total 
entropy production as 

^"«^^"-''>-''^'-wL,,/ll??)L*^ (31) 

The first term on the right hand side signifies a change in p(x, r) which can be due 
to a time-dependent A(r) or, even for a constant Ao, due to relaxation from a non- 
stationary initial state pq{x) ^ p''{x,Xo). 

As a variant on the trajectory level, occasionally (j){x, A) = — hip'^{x, A) has been 
suggested as a definition of system entropy. Such a choice is physically questionable 
as the following example shows. Consider diffusive relaxation of a localized initial 
distribution po{x) in a finite region < a; < L. Since p*(x) = 1/i, 0(a;) will not 
change during this process. On the other hand, such a diffusive relaxation should 
clearly lead to an entropy increase. Only in cases where one starts in a NESS and 
waits for final relaxation, the change in system entropy can also be expressed by a 
change in the non-equilibrium potential according to As = A(f). 



2.5. Ensemble averages 

Upon averaging, the expressions for the thermodynamic quantities along the individual 
trajectory should become the ensemble quantities of non-equilibrium thermodynamics 
derived previously for such Fokker-Planck systems, see, e.g. [19]. 

Averages for quantities involving the position x{t) of the particle are most easily 
performed using the probability p{x,t). Somewhat more delicate are averages over 
quantities like the heat that involve products of the velocity x and a function g{x). 
These can be performed in two steps. First, one can evaluate the average {x\x,t) 
conditioned on the position x in the spirit of the Stratonovitch discretization as 

{x\x,t) = lim {{x{t + At) — x{t)\x{t) ^ x) + 

Ar— >0 

(x(r) - x{t - At)\x{t) = x)) /(2Ar). (32) 

The averages in the bracket on the rhs can be evaluated by discretizing the path 
integral (4, 5) for one step. The first term straightforwardly yields ij,F{x,t)At. In 
the second one, the end-point conditioning (best implemented by Bayes' theorem) is 
crucial which leads to an additional contribution if the distribution is not uniform. 
The final result is [20] 

{x\x, t) = ^iF{x, t) - Ddxp{x, t) = iy{x, r). (33) 

Any subsequent average over position is now trivial leading to 

{g{x)x) = {g{x)iy{x, r)) = J dxg{x)j{x, t). (34) 

With these relations, one obtains, e.g., for the averaged total entropy production 
rate from (31) the expression 

^*°'(r) ^ {s'°\t)} = I dx^^^f^ = Hx,t)^)/D > 0, (35) 
J Dp(x, t) 
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where equality holds in equilibrium only. In a NESS, i'{x,t) — v'^{x) which thus 
determines the mean dissipation rate. Averaging the increase in entropy of the medium 
along similar lines leads to 

5'°(t) = = J dxF{x,T)3{x,T)/T. (36) 

Hence upon averaging, the increase in entropy of the system proper becomes S{t) = 
(s(r)) = S^°'^{t) — S™{t). On the ensemble level, this balance equation for the averaged 
quantities can also be derived directly from the ensemble definition of the system 
entropy 

S{t) = — j dx p{x, t) lnp(a;, r) — (37) 
by using the Fokker-Planck equation (7). 



2.6. Simple generalizations 

2.6.1. Underdamped motion For some systems, it is necessary to keep the inertial 
term which leads with mass m and damping constant 7 to the Langevin equation 

m£ + jx=-d^V{x,X) + fW+^ (38) 

with the noise correlations {£.{t)^{t')) = 2'-fTS{T — t'). 

The internal energy now must include the kinetic energy, dE ~ dV + mv dv, with 
V = X. Since the identification of work (15) remains valid, the first law becomes 

dq = dW -dE = Edx - mv dv. (39) 

Evaluating the stochastic entropy 

s(t) = - \np{x{T), v{t),t) (40) 

now requires a solution of the corresponding Fokker-Plank equation 

drP = -d^vp) - a„[[(-7« + F)/m]p - T{^/m^)d,,]p (41) 

for p ~ p{x, V, t) with an appropriate initial condition pq{x, v). 



2.6.2. Interacting degrees of freedom The framework introduced for a single degree of 
freedom can easily be generalized to several degrees of freedom x obeying the coupled 
Langevin equations 

i = ^[-Vl/(x,A) +f(x,A)] +C, (42) 

where ^(x. A) is a potential and f (x. A) a non-conservative force. The noise 
correlations 

(C(r):C(r'))=2T^<5(r-r'), (43) 

involve the mobility tensor fi. Simple examples of such system comprise a colloidal 
particle in three dimensions, several interacting colloidal particles, or a polymer where 
X labels the positions of monomers. If hydrodynamic interactions are relevant, the 
mobility tensor will depend on the coordinates x. 
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The corresponding Fokker-Planck equation becomes 

a^p(x, r) = - Vj = - V(/i(- + f )p - T/£Vp) (44) 

with the local (probability) current j. 

In particular for a NESS, there are two major formal differences compared to 
the one-dimensional case. First, the stationary current j''(x) becomes x dependent 
and, second, the stationary distribution p''(x) = exp[— 0(x)] or, equivalently, the non- 
equilibrium potential are not known analytically except for the trivial case that the 
total forces are linear in x. 

On a formal level, all expressions discussed above for the simple colloidal particle 
can easily be generalized to this multi-dimensional case by replacing scalar operations 
by the corresponding vector or matrix ones. 

2.6.3. Systems in external flow So far, we have assumed that there is no overall 
hydrodynamic flow imposed on the system. For colloids, however, external flow is 
a common situation. Likewise, as we will see, colloids in moving traps can also be 
described in a co-moving frame as being subject to some flow. We therefor recall 
the modifications required for the basic notions of stochastic thermodynamics in the 
presence of external flow field u(r) [59] . 

The Langevin equations for fc = 1, .., coupled particles at positions r^. reads 

rfe=u(r,)-f ^^^^(-V,V^ + fi)+Cfe (45) 

with the usual noise correlations 

{Ur)Ci{T') = 2TgJ{T-T')). (46) 

In such a system, the increments in external work and dissipated heat are given by 
dw =i\dr + u(r,)Vfc]y + h[vk - u(r,)]) dt (47) 

and 

dq^dw-dV^ {[Tk - u(r J]hVfel/ + ffc]) dt, (48) 

respectively. Compared to the case withour flow, the two modifications involve 
replacing the partial derivative by the convectivc one and measuring the velocity 
relative to the external flow velocity. These expressions guarantee frame invariance of 
stochastic thermodynamics [59]. 

For the experimentally studied case of one-dimensional colloid motion in a flow 
of constant velocity u discussed in Sect. 5.2.2 below, the Langevin equation simplifies 
to 

x = u + ^i{-d,V + f) + C (49) 
and the ingredients of the first law become 

dw = {drV + ud^V)dt + /(i - u)dt (50) 



and 



dq= {x - u)[-d:,V + f] dt. 



(51) 
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3. Fluctuation theorems (FTs) 

Fluctuation theorems express universal properties of the probability distribution 
p{VL) for functionals f2[a;(r)], like work, heat or entropy change, evaluated along the 
fluctuating trajectories taken from ensembles with well-specified initial distributions 
Pq{xq). In this section, we give a phenomenogical classification into three classes 
according to their mathematical appearance and point out some general mathematical 
consequences. The most prominent ones will then be discussed in physical terms with 
references to their original derivation. For proofs of these relations within stochastic 
dynamics from the present perspective, we provide in Sect. 4 the unifying one for all 
FTs that also shows that there is essentially an infinity of such relations. 



3.1. Phenomenological classification 

3.1.1. Integral fluctuation theorems (IFTs) A non-dimcnsionalized functional 
r2[x(r)] with probability distribution function p(ri) obeys an IFT if 



(cxp(-t7)) = j dn cxp(-17) = 1. (52) 

The convexity of the exponential functions then implies the inequality 

> (53) 

which often represents a well-known thermodynamic law. With the exception of the 
degenerate case, p{^) = the IFT implies that there are trajectories for which VI 

is negative. Such events have sometimes then been characterized as "violating" the 
corresponding thermodynamic law. Such a formulation is controversial since classical 
thermodynamics, which ignores fluctuations from the very beginning, is silent on issues 
beyond its range of applicability. The probability of such events quickly diminishes 
for negative f2. Using (52), it is easy to derive for w > [60] 

dn p{n) e~'^-" < e-'^. (54) 

-oo 

This estimate shows that relevant "violations" occur for O of order 1. Restoring the 
dimensions in a system with N relevant degrees of freedom, will typically be of order 
NksT which implies that in a large system such events are exponentially small, i.e., 
occur exponentially rarely. This observation essentially reconciles the effective validity 
of thermodynamics on the macro-scale with the still correct mathematical statements 
that even for large systems, in principle, such events must occur. 

An IFT represents one constraint on the probability distribution p{fl). If it is 
somehow known that p{ft) is a Gaussian, the IFT implies the relation 

{{n - = 2{n) (55) 

between variance and mean of fl. 



3.1.2. Detailed fluctuation theorems (DFTs) A detailed fluctuation theorem 
corresponds to the stronger relation 



p{-n)/p(Sl) = exp(-17) 



(56) 
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for the pdf Such a symmetry constrains "one half" of the pdf which means, 

e.g., that the even moments of 51 can be expressed by the odd ones and vice versa. 
A DFT imphes the corresponding IFT trivially. Further statistical properties of p(il) 
following from the validity of the DFT (and some from the IFT) are derived in [61]. 

Depending on the physical situation, a variable obeying the DFT has often been 
called to obey either a transient FT (TFT) or a steady state FT (SSFT). These notions 
will be explained below for the specific cases. 

3.1.3. (Generalized) Crooks fluctuation theorems (CFTs) These relations compare 
the pdf p(f2) of the original process one is interested in with the pdf (17) of the same 
physical quantity for a "conjugate" (mostly the time-reversed) process. The general 
statement then is that 

p\-n)=p{n)e-'^ (57) 
which implies the IFT (but not the DFT) for VL since is normalized. 

3.2. N on- equilibrium work theorems 

These relations deal with the probability distribution p{'w) for work spent in driving 
the system from a (mostly equilibrium) initial state to another (not necessarily 
equilibrium) state. They require only a notion of work defined along the trajectory 
but not yet the concept of stochastic entropy. 

3.2.1. Jarzynski relation (JR) In 1997, Jarzynski showed that the work spent in 
driving the system from an initial equilibrium state at Aq via a time-dependent 
potential V{x, A(t)) for a time t obeys [6] 

(exp(-w/T)) = exp - (AJ^/T) (58) 

where = J^{Xt) — -^(Aq) is the free energy difference between the equilibrium state 
corresponding to the final value At of the control parameter and the initial state. In 
the classification scheme proposed here, it can technically be viewn as the IFT for the 
(scaled) dissipated work 

Wd = {w~ AT)/T. (59) 

The paramount relevance of this relation - and its originally so surprising feature - 
is that it allows to determine the free energy difference, which is a genuine equilibrium 
property, from non-equilibrium measurements (or simulations). It represents a 
strengthening of the familiar second law (w) > AF which follows as the corresponding 
inequality. It has orginally been derived using a Hamiltonian dynamics (which requires 
decoupling from and coupling to a heat bath at the beginning and end of the process, 
respectively) but soon been shown to hold for stochastic dynamics as well [7, 8, 9]. 
Its validity requires that one starts in the equilibrium distribution but not that the 
system has relaxed at time t into the new equilibrium. In fact, the actual distribution 
at the end will be p{x, t) but any further relaxation at constant A would not contribute 
to the work anyways. 

Within stochastic dynamics, the validity of the JR (as of any other FT with 
a thermodynamic interpretation) essentially rests on assuming that the noise in the 
Langevin equation (1) is not affected by the driving. A related issue arises in the 
Hamiltonian derivation of the JR which requires some care in identifying the proper 
role of the heat bath during the process [62, 63]. 
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The JR has been studied for many systems analytically, numerically, and 
experimentally. Specific case studies for stochastic dynamics will be classified and 
quoted below in Sect. 5. As an important application, based on a generalization 
introduced by Hummer and Szabo [10], the Jarzynski relation can be used to 
reconstruct the free energy landscape of a biomolecule as discussed in Sect. 9.3 below. 

3.2.2. Bochkov-Kuzolev relation (BKR) The Jarzynski relation should be 
distinguished from an earlier relation derived by Bochkov and Kuzolev [11, 12]. For a 
system initially in equilibrium in a time- independent potential Vo{x) and for < t < t 
subject to an additional space and time-dependent force (possibly arising from an 
additional potential), the work (25) integrated over a trajectory obeys the Bochkov- 
Kuzolev relation (BKR) 

(exp[-u''='^/T]) = 1. (60) 

Contrary to some claims, the BKR is different from the Jarzynski relation since they 
apply a priori to somewhat different situations [21, 64, 65]. The JR as discussed above 
applies to processes in a time-dependent potential, whereas the BKR applies to a 
process in a constant potential with some additional force. If, however, in the latter 
case, this explicit force arises from a potential as well, both the BKR and the JR (58) 
hold for the respective forms of work. 

3.2.3. Crooks fluctuation theorem (CFT) In the Crooks relation, the pdf for work 
p{w) spent in the original (the "forward") process is related to the pdf for work 
p{w) applied in the reversed process where the control parameter is driven according 
to A(t) = A(t — r) and one starts in the equilibrium distribution corresponding to 
Ao — \t- These two pdfs obey [8, 9] 

f>{~-w)lp{w) = exp[-(u> - AJ")/T]. (61) 

Hence, AJ" can be obtained by locating the crossing of the two pdfs which for 
biomolecular applications turned out to be a more reliable method than using the 
JR. Clearly, the Crooks relation implies the JR since p{w) is normalized. Technically, 
the Crooks relation is of the type (57) for R ^ Wd with the conjugate process being 
the reversed one. 

3.2.4. Further general results on p{w) Beyond the JR and the CFT, further exact 
results on p{w) are scarce. For systems with linear equation of motion, the pdf for 
work (but not for heat) is a Gaussian for arbitrary time-dependent driving [66, 67]. 
For slow driving, i.e., for tj-d/t <^ 1 where tj-ai is the typical relaxation time of the 
system at fixed A and t the duration of the process, an expansion based on this time- 
scale separation yields a Gaussian for any potential [68] . Such a result has previously 
been expected [69, 70] or justified by invoking arguments based on the central limit 
theorem [71]. Two observations show, however, that such an expansion is somewhat 
delicate. First, even in simple examples there occur terms that are non-analytic in 
troi/t [68]. Second, for the special case of a "breathing parabola" , V{x, A) = X{t)x^ /2, 
any protocol with A > leads to p{w) = for w < which is obviously violated by a 
Gaussian. How the latter effectively emerges in the limit of slow driving is investigated 
in [72]. 

From another perspective, Engel [73, 74] investigated the asymptotic behavior of 
p{w) for small T using a saddle point analysis. The value of this approach is that 
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it can provide exact results for the tail of the distribution. Specific examples show 
an exponential decay. Saha et al. [75] suggest that the work distribution for quite 
different systems can be mapped to a class of universal distributions. 

3.3. FTs for entropy production 

3.3.1. IFT The total entropy production along a trajectory as given by 

As*°' = As'" + As, (62) 

with 

As = - lnp(a;t, At) + lnp(xo, Ao) (63) 
and As"^ defined in (27), obeys the IFT [20] 

(exp(-As'°*)) = 1 (64) 

for arbitrary initial distribution p(x, 0), arbitrary time-dependent driving A(t) and an 
arbitrary length t of the process. 

Formally, this IFT can be considered as a refinement of the second law, (As*°') > 
0, which is the corresponding inequality. Physically, however, it must be stressed that 
by using the Langevin equation a fundamental irreversibility has been implemented 
from the very beginning. Thus, this IFT should definitely not be considered to 
constitute a fundamental proof of the second law. 

3.3.2. Steady-state fluctuation theorem (SSFT) In a NESS with fixed A, the total 
entropy production obeys the stronger SSFT 

p(-As'°')/p(As'°') = exp(-As'°') (65) 

again for arbitrary length t. This relation corresponds to the genuine "fluctuation 
theorem" . It has first been found in simulations of two-dimensional sheared fluids [1] 
and then been proven by Gallavotti and Cohen [2] using assumptions about chaotic 
dynamics. For stochastic diffusive dynamics as considered specifically in this review, 
it has been proven by Kurchan [3] and Lebowitz and Spohn [4]. Strictly speaking, in 
these early works the relation holds only asymptotically in the long-time limit since 
entropy production had been associated with what is here called entropy production 
in the medium. If one includes the entropy change of the system (28), the SSFT holds 
even for finite times in the steady state [20] . 

3.3.3. Hatano-Sasa relation The Hatano-Sasa relation applies to systems with steady 
states p^{x,X) = exp[—(j){x,X)]. With the splitting of the dissipated heat into a 
housekeeping and excess one (21), the IFT [13] 

(exp[-(A0 + g-/r)]) = l (66) 

holds for any length of trajectory with A(f> = 4>{xt, \t) — (f'i^o, ^o)- The corresponding 
inequality 

(A0) > -(g°'^)/T (67) 

allows an interesting thermodynamic interpretation. The left-hand side can be seen 
as ensemble entropy change of the system in a transition from one steady state to 
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another. Within the framework discussed in this review, this interpretation is hterahy 
true provided one waits for final relaxation at constant \t since then As = /S.(f). A 
recent generalization of the HS relation leads to a variational scheme for approximating 
the stationary state [76]. 

With the interpretation of the left hand side as entropy change in the system, 
the inequality (67) provides for transitions between NESSs what the famous Clausius 
inequality does for transitions between equilibrium states. The entropy change in the 
system is at least as big as the excess heat flowing into the system. For transitions 
between NESSs, the inequality (67) is sharper than the Clausius one (which still 
applies in this case and becomes just (As) > —q/T) since q scales with the transition 
time whereas q'^^ can remain bounded and can actually approach equality in (67) for 
quasistatic transitions. 

Experimentally, the Hatano-Sasa relation has been verified for a colloidal particle 
pulled through a viscous liquid at different velocities which corresponds to different 
steady states [77]. 

3.3.4- IPT for housekeeping heat Finally, it should be noted that the second 
contribution to heat, the housekeeping heat also obeys an IFT [78] 

(exp[-ghk/y]^ ^ ^ (gg) 
for arbitrary initial state, driving and length of trajecories. 

4. Unification of FTs 

Originally, the FTs have been found and derived on a case by case approach. However, 
it has soon become clear that within stochastic dynamics a unifying strategy is 
to investigate the behaviour of the system under time-reversal. Subsequently, it 
turned out that comparing the dynamics to its "dual" one [13, 79], eventually also in 
connection with time-reversal, allows a further unification. In this section, we outline 
this general approach and show how the prominent FTs discussed above (and a few 
further ones mentioned below) fit into, or derive from, this framework. Even though 
this section is inevitably somewhat technical and dense, it is self-contained. It could 
be skipped by readers not interested in the proofs or systematics of the FTs. For 
related mathematically rigorous approaches to derive FTs for diffusive dynamics, see 
[80, 81, 82, 83]. 

4.1. Conjugate dynamics 

FTs for the original process with trajectories a;(r), < r < i, an initial distribution 
Po(a^o) and a conditional weight p[a;(T)la;o] are most generally derived by formally 
invoking a "conjugate" dynamics for trajectories x^{t). These are supposed to obey 
a Langevin equation 

^ ^l^F''{x\\^) + C'^. (69) 

with {C{t)C{t')) = 2^i^T^S{t - r'). The trajectories with weight p'l'[a;'l'(T)]xJ] run 
over a time t and start with an initial distribution p^(a;J). Averages of the conjugate 
dynamics will be denoted by (...)^. 

This conjugate dynamics is related to the original process by a one-to-one mapping 

{x(T),A(r),F,//,T} ^ {x\T),XHT),F\fi\T^} (70) 
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which ahows to express aU quantities occuring in the conjugate dynamics in terms of 
the orginal ones. 

The crucial quantity leading to the FTs is a master functional given by the log- 
ratio of the unconditioned path weights 

i?[x(r)]^ln.^["(")] 



^ In ^ + In ZMll^ ^ i?o + i?i (71) 

that consists of a "boundary" term i?o coming from the two initial distributions and 
a "bulk" term Ri. 

Three choices for the conjugate dynamics and the associated mapping have been 
considered so far. In all cases, neither the temperature nor the functional form of the 
mobilities have been changed for the conjugate dynamics, i.e., T^ = T and /i^ = /i. 

(i) Reversed dynamics: This choice corresponds to "time-reversal" . The mapping 
reads 

x'' (t) = x{t - t) and X^t) = X{t - t) (72) 

with no changes at the functional dependence of the force from its arguments, i.e., 
Ft(xt,At) =F(a;t,At). 

The weight of the conjugate trajectories is easily calculated using the mapping 
(72) in the weight (4,5) leading to 

i?i - A{[x\t), At(r)]) - A{[x{t), A(r)])] = As^ = q/T, (73) 

which is the part of the action A{[x{t), X{t)]) that is odd under time-reversal. 

This relation allows a deep physical interpretation. For given initial point xq 
and final point Xj, the log ratio between the probability to observe a certain forward 
trajectory and the probability to observe the time-reversed trajectory is given by the 
heat dissipated along the forward trajectory. 

(a) Dual dynamics: This choice alters the equations of motion for the x^(r) 
trajectories such that (i) the stationary distribution remains the same for both 
processes and that (ii) the stationary current for the dual dynamics is minus the 
original one. Specifically, this mapping reads [79] 

F^x\X^) = F{x\X^) - 2v''ix\ At)//i (74) 

which enters the conjugate Langevin equation (69) and no modification for x and A, 
i.e., x''{t) = x{t) and A^(t) = A(r). 

Calculating the action for the dual dynamics (69), the functional i?i becomes 

Ri = q^'^/T = As^^. (75) 

(Hi) Dual-reversed dynamics: For this choice, the dual dynamics is driven with 
the time- reversed protocol, i.e., the mapping of the force (74) is combined with the 
time- reversal (72). In this case, the functional i?i becomes [79] 

Ri = q'^'^/T = As<='^. (76) 

In summary, depending on the form of the conjugate dynamics, different parts of 
the dissipated heat form the functional Ri. For later reference, we have introduced 
in the last two equations for the scaled contributions to the dissipated heat the 
corresponding entropies. 
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^.2. The master FT 

4-2.1. Functionals with definite parity The FT's apply to functionals S'Q[a;(T)] of the 
original dynamics that map with a definite parity Eq, = ±1 to the conjugate dynamics 
according to 

Si{[xHT)lX\F^)=e^So.MT)],X,F) (77) 

such that 5^[a;^(r)] represents the same physical quantity for the conjugate dynamics 
as Sa[x{T)] does for the original one. 

Examples for such functionals are work and heat that both are odd (ea = — 1) 
for the reversed dynamics. For dual or dual reversed dynamics, however, these two 
functionals have no definite parity since both cases involve a different dynamics. 
Explicitly, the heat behaves under time- reversal as = drx'^ F'^ ~ ~ Iq ^txF = — g. 
For dual dynamics, the heat transforms as = J^drx^F^ = J^dTx{F — 2v^/fj,) = 
q — 2q^^^ which has, in general, no definite parity. On the other hand, the housekeeping 
heat is odd for the dual dynamics and even for both the reversed and the dual-reversed 
dynamics. 

The stochastic entropy As, in general, has no definite parity under time- reversal 
since s(r) is defined through the solution p{x, r) of the Fokker-Planck equation which is 
not odd under time- reversal. In particular, p{x, t — r) does not solve the Fokker-Planck 
equation for the time-reversed process even if one starts the reversed process with the 
final distribution p(x, t) of the original process. The change in the non-equilibrium 
potential Ac/), however, is odd under time-reversal. This difference between As and 
A(j) implies that Ac/) occurs more frequently in FTs. 



4.2.2. Proof With these preparations, one can easily derive the master FT 
= I dxll d[x\T)]pl{xl)p[x\T)\xl]g{{e^Si}) 



dxl / d[x\T)]pQ{xQ)p[x{T)\xQ\cx:p[~R]g{{Sa}) 



= J dxQ J d[x{T)]pQ{xo)p[x{T)\xo]eyi-p[-R]g{{Sa}) 

= (.9({5„[x(r)]})exp(-i?[x(r])) (78) 

for any function g depending on an arbitrary number of such functionals Sa- For the 
second equality, we use the definitions (71) and the parity relation (77); for the third 
we recognize that summing over all daggered trajectories is equivalent to summing 
over all original ones both for x^{t) = x{t) and x'^{t) = x{t — r). With the choice 
g = 1, this FT leads to the most general IFT (e^^) = 1 from which all known IFT-like 
relations follow as shown in Sect. 4.3. 

By choosing for g the characteristic function, one obtains a generalized FT for 
joint probabilities in the form 

pHjSi = eas^}) ^ ^ exp(-i?)|{54 = {sj) (79) 



that relates the pdf for the conjugate process to the pdf of the original one and 
a conditional average. All known DFTs for stochastic dynamics follow as special 
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cases of this general theorem as shown in Sect. 4.4. The key point is to (i) select 
the appropriate conjugate process for which the quantity of interest has a unique 
parity, which is most often just the reversed dynamics, (ii) identify for the generally 
free initial distribution p\f(x) an appropriate function, and (iii) express the functional 
R using physical quantities, preferentially the quantity of interest il. 



4.3. General IFTs 

The simplest choice for the function g in (78) is the identity, g = 1, leading to the IFT 
(e^^) = 1. Explicitly, one obtains for the three types of conjugate dynamics: 
(i) By choosing the reversed dynamics (72) and with (73), the class of IFTs 

^exphA.-]\=l (80) 

follows for any initial condition po (2^0)1 a-ny length of trajectories t, and any normalized 
function •p\(xt) = Po(a;t) [20]. By specializing the latter to the solution of the Fokker- 
Planck equation for r = t one obtains the IFT for total entropy production (64). 

For a system in a time-dependent potential Vix^ A) and by starting in an initial 
distribution given by the corresponding Boltzmann factor, Pq{x) = cxp[— (l/(a;, Aq) — 
J'(Ao))/T], one obtains the JR (58) for the choice pi(a;t) exp[-(X^(a-, \t)~T(\t))lT\ 
corresponding to the Boltzmann distribution for the final value of the control 
parameter. 

A variety of "end-point" relations can be generated from (80) as follows. By 
choosing pi(a;) = p(x,t)g(x) / {g(xt)) , one obtains 

(g(xOexp[-A,s*°']) = (g(xO) (81) 

for any function g{x) [84]. Likewise, for / = and F(a;, A(t)), by choosing 
pi{x) = g{x)c^-p[-{V{x, At) - F{\t))/T]l {g{x)Y-^^, one obtains 

{g{xt) exp[-(u; - A^)/T]) = {g(x))2 (82) 

which has been first derived by Crooks [9]. Here, the average on the right hand side 
is the equilibrium average at the final value of the control parameter. In the same 
fashion, one can derive 

(g(xOexp[~«;-/T]) = (g(.x))°^ (83) 

by choosing pi{x) = 5(0:) exp[-(\/(x, Aq) - •7^(Ao)))/T]/(g(a;))^^ for a time- 
independent potential and arbitrary force f{x, r) which is the end-point relation 
corresponding to the BKR (60). The latter follows trivially by choosing g{xt) — 1. 

For processes with feedback control as discussed in Sect. 7.3 below, it will be 
convenient to exploit the end-point conditioned average 

cxp[-As'"] I xt = x)p{x, t) = 1 (84) 



Pq{x()) 



valid for any x which follows from (80) by choosing pi{xt) = 5{xt — x). Equivalently, 
by choosing pi(x, i) =p{x,t), 

J dxo(exp[-As'>(a;t,OI-^o) = 1 (85) 
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holds for summing over the initial point conditioned average. 

(ii) By using the dual dynamics with pJ(a;o) = Pa{xa)^ the IFT for the 
housekeeping heat [78] 

(exp[-ghk/T]) = 1 (86) 

valid for any initial distribution follows. 

(iii) For the dual-reversed dynamics, one gets the class of IFTs from 

/ElMexp[-g-/T]\ = l (87) 
\Po(a;o) / 

valid for any initial distribution pq{xq) and any normalized function pi{xt). By 
choosing pq(xq) = exp[— 0(.to, Ao)] and pi(xt) = exp[— ^(xt, A*)], one obtains the 
Hatano-Sasa relation (66). Similarly, another class of end-point relations could be 
generated starting from (87). 

Finally, since the IFTs, (exp[— fi]) = 1, do not explicitly involve the conjugate 
process, one might wonder whether they can be derived in an alternative way. Indeed, 
some of them can be obtained by deriving an appropriate Fokker-Planck-type equation 
for the joint pdf p(r2, x, r) and then showing dr{e~^) = dr J dil J dx e~^p{V,, x, r) = 
directly, see for the Jarzynski relation [7], for the house- keeping heat [78], and for 
another large class of IFTs [79] . 



4.4- FTs derived from time-reversal 

In this section, the FTs following from using time-reversal as conjugate dynamics are 
derived systematically from (79) by specializing to the various scenarii concerning 
initial conditions and type of driving. More or less reversing the chronological 
development, we start with the more general cases and end with the more specific 
ones, for which the strongest constraints on these pdfs follow. 



4.^.1. CFTs involving reversed dynamics By starting both, original and reversed 
dynamics, in the respective stationary state, the functional R becomes 

i? = A0 + As"\ (88) 

Hence, one gets from (79) the FT 

p^iSl^e^s^) ^ ^ ^_(^,^.^,»)|^^^^ ^ ^^^^^^ ^gg^ 

For the special case that Sa ~ A(/) -I- As™, this relation has first been derived by 
Garcia-Garcia et al. [85]. Note that in general the change in stochastic entropy As is 
not an admissible choice for Sa since it lacks definite parity under time-reversal. 
By choosing for Sa the work w, one gets 

p\~w)^ p{w) (e-('^'^+'^^") \w) . (90) 

From this relation, the Crooks FT (57) follows for a time-dependent V{x, A(r)) and 
/ = 0, if one samples both processes from the respective initial equilibria, since then 
A(f> A(y - T)/T and hence R = {w - AT)/T. 

Likewise, by choosing Sa — w XA{xo)Xb{xt), where xa,b = 1(0) if a; S {^)A, B are 
the characteristic functions of two regions A and B, one obtains the variants derived 
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and discussed in [86, 87] which allow to extend the CFT to "partially equilibrated" 
initial and final states. These variants have become useful in recovering free energy 
branches in single molecule experiments. 

As another variant, by choosing for Sa the work w'^^ and by starting the reverse 
process in the initial equilibrium, one gets with A(/) ~ ISV^ and R = (AF" + q)/T = 
w"^^ the Crooks relation for w'^^ [65] 

= exp[-w°'^/r]. (91) 

4-4-2- DFTs for symmetric and periodic driving For symmetric driving, A(t) — 
\{t — t). and for = Pol^^o); the reversed dynamics becomes the original one. 

Hence, the FTs (89,90) derived in the previous subsection remain valid in this case if 
one replaces on the left hand side with p. In this case, as in those in the following 
subsections, the FTs no longer involve the conjugate dynamics explicitly which thus 
has become a mere mathematical tool to derive these relations most efficiently. In 
particular, for starting in the initial equilibrium, R — w/T and one gets for the pdf of 
work [88, 89] 

p{-w)/p{w) = exp[-u;/r]. (92) 

Likewise, for a periodically driven system with an integer number of symmetric 
periods of length tp, i.e., \{tp — t) = A(t), the reversed dynamics is the original one. 
If the distribution has settled into a periodic stationary state, one has the DFT-like 
relation for total entropy production [90, 91]. Note that it is crucial to choose not 
only a periodic but also a symmetric protocol since otherwise the reversed dynamics 
is not the original one. 



4-4-3. SSFTs for NESSs For a NESS, i.e., for time-independent driving and starting 
in the stationary state, the reversed dynamics becomes the original one and thus 
R = As*°*. Then (79) implies the generalized SSFT for joint probabilities in the form 

For this case, system entropy is indeed odd, and hence one also has, in particular, by 
choosing Sa = As'°* the genuine SSFT (65) for total entropy production and arbitrary 
length t. As variants, illustrating the potency of the general theorem, one easily gets 
from (93) 

p(-A.s) ==p(As)e-^"(e-^''"'|As) (94) 

and 

p(-As'") = p(As™)e-'^''"(e-^"|As™) (95) 

involving conditional averages. Such relations seem not to have been explored in 
specific systems yet. 



4-4-4- Expression for the NESS distribution By using an initial and end-point 
conditioned variant of (79), Komatsu et al. manage to express the stationary 
distribution p^{x) in a NESS by non-linear averages over the difference in "excess" 
heat required either to reach x from the steady state or to reach the NESS starting 
in X [92, 93, 94] which leads to Clausius-type relations for NESSs [95]. For a related 
expression for p"^ (x) in terms of an expansion around a corresponding equilibrium state, 
see [96] which contains a valuable introduction into the history of such approaches. 
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4.4-5. Transient fluctuation theorems TFTs This relation applies to time- 
independent driving and arbitrary initial condition pq(xq). If the reversed dynamics 
is sampled using the same initial condition, Pq{xq) = ^0(2^0)7 then the functional R 
becomes 

i? = - ln[po{xt)/poixo)] + q/T = Qt (96) 

which has been called dissipation functional by Evans and Searles [37]. Physically, ilt 
corresponds to the log-ratio between the probability to observe the original trajectory 
and the one for observing the time-reversed one. Since under these conditions fit is 
odd and the reversed dynamics is equivalent to the original one, one has from (79) the 
TFT 

p{-nt)/pi^lt) - c^-^t] (97) 

valid for any length t and initial condition. 

If the system is originally equilibrated in a potential Vo{x) and then suddenly 
subject to either another time- independent potential Vi{x) or a force f{x) the 
dissipation functional becomes 

^It = Vo{xt) - Viixt) + Viixo) - Voixo), (98) 

or fif = w, respectively. In the latter case, the TFT holds for work. 

4-5. FTs for variants 

4-5.1. FTs for underdamped motion For underdampcd motion as introduced in Sect. 

2.6.1. the functional Ri defined in (71) under time-reversal is still given by the 
dissipated heat, i.e., by (39) integrated over the trajectory [97]. This fact follows by 
directly evaluating the action for the path integral corresponding to the underdamped 
Langevin equation (38). Hence, all FTs based on time- reversal hold true also for 
underdamped dynamics with the obvious modification that initial (and daggered) 
distributions now depend on x and v. 

4.5.2. FTs in the presence of external flow In the presence of flow, one has to specify 
how the flow changes in the conjugate dynamics. For genuine time-reversal, the 
physically appropriate choice is u(r)^ = ^u(r) which leads with the definitions of 
work and heat (47,48) to an odd parity for these two functionals. Consequently, the 
FT's then hold as in the case without flow. Formally, however, one could also keep the 
flow unchanged for the conjugate dynamics, u(r)^ = u(r), which would lead to another 
class of FTs. For a specific example illustrating this freedom, see the discussion in 
[59] for a dumbell in shear flow first investigated in [98] . 

4.5.3. FT with magnetic field In the presence of a (possibly time-dependent) 
magnetic field, the FTs hold true essentially unchanged as proven in great generality 
for the JR and the CFT for interacting particles on a curved surface [99]. This work 
generalized earlier case studies on the validity of the JR for specific sitations involving 
a magnetic field as mentioned in Sect. 5.2 below. A second motivation for this work 
was to refute earlier claims based on simulations that the Bohr-van-Leeuven theorem 
stating the absence of classical diamagnetism could fail for a closed topology [100]. 
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4-5. 4- Further "detailed theorems" Esposito and van dcr Brocck have derived what 
they call detailed fluctuation theorems for As*°*, As*^"*^, (called "adiabiatic" entropy 
change As'*'^) and for the "non-adiabiatic" entropy change As"'' = As°^ + As under 
even more general conditions [101, 102, 103]. Their relations are beyond the realm 
of the present systematics since they compare the pdf for different physical quantities 
for the original and the conjugate process whereas we always compare the pdfs of the 
same physical quantities. § A unification of FTs within this broader sense involving 
joint distributions of these decompositions of entropy production is achieved in [104] 
and a generalization of the Hatano-Sasa relation in [105]. 



4-6. FTs for athermal systems 

4-6.1. General Langevin systems The derivation of the master FT in Sect. 4.2 shows 
that for obtaining these mathematical relations the main requirement is the existence 
of a conjugate dynamics such as time reversal. Therefore, imposing a relation between 
the strength of the noise and the mobility as done in Sect. 2.1 for colloidal particles is 
not really necessary. Neither is it necessary to interpret the Langevin equation using 
concepts of work and heat. We therefore sketch in this section the general FT for a 
system of Langevin equations 

i = K(x,A)+C (99) 
with arbitrary "force" K and noise correlations 

(C(T):C(r')) = 2^<5(r-r'). (100) 

The corresponding Fokker-Planck equation becomes 

9^]9(x, r) = - Vj = - V(Kp - ^Vp) (101) 

and the local (probability) current j(x. A). For constant A, one has the local mean 
velocity 

v^(x, A) = j^(x, A)/p^(x, A) = K(x, A) - (102) 

For time reversal as conjugate dynamics, by evaluating the corresponding weight 
one obtains for the master functional (73) 

i?i = / dT±D-^K = As"" (103) 



where the identification with As™ is now purely formal. If one adds the stochastic 
entropy change along a trajectory 

As EE - ln[p(xt, At)/p(xo, Ao)] (104) 

one obtains the total entropy production As'°'. Likewise, in analogy to the colloidal 
case, As™ can be split into 

As''''= [ dTxUr\' (105) 

and ^ ^ 

As'''' = As™ - As'^'^ = / drxVlnp" = -A0+ / dAA^A^. (106) 
Jo Jo 

§ The IFT {exp[— As"'']) = 1, however, follows directly from (87) by choosing pi{xt) = p{x, t). 
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With these identifications ah FTs involving the various forms of entropy production 
derived and discussed in Sects. 3.3 and 4 hold true for such Langevin systems as well. 

The identification of a generalized work makes immediate sense only if K = 
— (Z2/T) Vy(x, A) with some potential y(x, A) and effective temperature T in which 
case one is back at the thermal model with interacting degrees of freedom introduced 
in Sect. 2.6.2. If K cannot be derived in this way from a gradient field there seems 
to be no gain by trying to impose a genuine thermodynamic interpretation without 
further physical input. 

4-6.2. Stochastic fields The generalization of the results in the previous section for 
coupled Langevin equations to stochastic field equations is trivially possible [21]. 
Consider a scalar field 4'(r,T) that obeys 

9,*(r, r) = A-[*(r, r), A(t)] + C(r, r) (107) 

with some functional K[^{r, r), A(t)] and 

(C(r, r)C(r', r')) = 2D{r - r')<5(r - r'). (108) 

with arbitrary spatial correlation D{{r — r'). The expressions for the entropy terms 
can easily be inferred; e.g., the analogy of the entropy change in the medium becomes 

As^ss^ dr j dr J dr'dr'if{r,T)D-\r - r')K['^{r' ,t), X{t)]. (109) 

By now, it should be obvious how to derive the corresponding FTs and how to 
generalize all these also to the case when ^(r, r) is a multi-component field. Likewise, 
it would be a trivial task to specialize all this to driven or relaxing "thermal" field 
theories for which K includes the derivative of some Landau-Ginzburg type free energy 
and where the noise obeys an FDT [106]. 

An interesting applications concerns enstrophy dissipation in two-dimensional 
turbulence [107]. Field-theoretic techniques are used in [108] to derive generahzed 
JRs and to explore the role of super-symmetry in this context. Quite generally, it will 
be interesting to investigate stochastic versions of the field equations of active matter 
[109] from this perspective. 

4.6.3. FTs in evolutionary dynamics The framework of FTs has recently been 
applied to the stochastic evolution of molecular biological systems where it leads to 
an IFT for fitness fiux [110]. 

5. Experimental, analytical and numerical work for specific systems with 
continuous degrees of freedom 

5.1. Principal aspects 

The various relations derived and discussed above have the status of mathematically 
exact statements. As such they require neither a "test" nor a "verification". The 
justification for, and the value of, the large body of experimental and numerical work 
that appeared in this field during the last decade rather arises from the following 
considerations. 
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First, experimental and numerical measurements of the distributions p{^) 
entering the theorems provide non-trivial information about the specific system under 
consideration. Integral and detailed theorems give only one constraint on, and 
constrain only one half of, the distribution, respectively. Beyond the constraints 
imposed by the exact relations, the distributions are non-universal in particular for 
short times. 

Second, the theorems involve non-linear averages. The necessarily limited number 
of data entering experimental or numerical estimates can cause deviations from the 
predicted exact behavior. It is important to get experience how large such statistical 
errors are. Systematic theoretical investigations concerning the error due to finite 
sampling are mentioned in Sect. 9.3 below. 

Third, the thermodynamic interpretation of the mathematical relations in terms 
of work, heat and entropy rests on the crucial assumption that the noise in the 
Langevin equation is not affected by the driving. While this condition can trivially 
be guaranteed in simulations, it could be violated in experiments. A statistically 
significant deviation of experimental results from a theoretical prediction could be 
rooted in the violation of this assumption. 

These remarks apply to systems where one expects at least in principle that a 
stochastic description of the relevant degrees of freedom well-separated in time-scale 
from an equilibrated heat bath is applicable. There are, however, systems that a 
priori do not belong to this class like sheared molecular fluids, shaken granular matter 
and alike. The proof of fluctuation theorems given above will not apply to such 
systems. Still, fluctuation theorems have been proved for other types of dynamics and 
experimentally investigated in such systems. 

In the following we first focus on a review of experimental and numerical work of 
the first category and then briefly mention systems for which it is less clear whether 
they comply with the assumptions of a stochastic dynamics. When refering to the 
experiments and the numerical work, we will use the notions and notations established 
in this review, which may occasionally differ from those given by the original authors. 

5.2. Overdamped motion: Colloidal particles and other systems 

5.2.1. Equilibrium pdf for heat Even in equilibrium, explicit calculations of the pdf 
for heat arc typically non-trivial. In the long time t — > oo, low temperature T — > 
limit, it has been calculated for an arbitrary potential with multiple minima [111]. For 
a harmonic potential and any t and T, it is given by an expression involving a Bessel 
function [112]. It has also been derived analytically in the presence of a magnetic field 
[113]. 

5.2.2. Moving harmonic traps and electric circuits Wang et al. [114] measured the 
distribution of what amounts to work (called Et in their eq. 2) for a colloidal particle 
initially in equilibrium in a harmonic trap which was then displaced with constant 
velocity. The authors found that the pdf obeys a relation corresponding to the TFT 
which is strictly speaking the correct interpretation only within the co-moving frame. 
Interpreted in the lab frame, the driving is time-dependent. However, since for linear 
forces the work distribution is a Gaussian which moreover has to obey the JR with 
AT = 0, it is clear that such a Gaussian also obeys the TFT formally. 

In a sequel, Wang ct al. [115] considered the same set-up for a quasi-steady state 
situation at constant velocity. The authors showed in particular that a quantity (fit (r) 
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as defined in their eq. 19) which is equal to As*°' obeys the DFT also for short times 
as it should sinee this set-up seen in the co-moving frame corresponds to a genuine 
NESS. 

For traps moving with constant velocity, exphcit expressions for the Gaussian 
work distribution, i.e. for its mean and variance, have been calculated in [66, 116]. In 
all cases, the DFT type relation is fulfilled. In contrast, the pdf for the dissipated heat 
is non Gaussian (essentially because it involves the distribution of (xt — At)^ from the 
internal energy) with exponential tails. An explicit expression is not available, but its 
characteristic function and in consequence its large deviation form can be determined 
analytically [117, 118]. The pdf for work in a moving trap (and the pdf for heat in a 
stationary trap) were also measured and compared to theoretical results by Imparato 
et aJ. [119]. For a harmonically bound particle subject to a time-dependent force, Saha 
et al. calculated pdfs for total entropy production, in particular, for non-equilibrated 
initial conditions [120]. 

For a charged particle in a harmonic trap, work fluctuations and the JR have 
been studied theoretically for a time-independent magnetic field and a moving trap 
or time-dependent electric field in [121, 122, 123, 124, 125], and for a time-dependent 
magnetic field in [126], respectively. 

Trepagnier et al. [77] studied experimentally the transition from one NESS to 
another by changing the speed of the moving trap. If interpreted in the co-moving 
frame, their experiment constituted the first experimental verification of the Hatano- 
Sasa relation (66). 

Simple electric circuits can formally be mapped to the dragged colloidal particle. 
Corresponding FTs and pdfs have been investigated by Ciliberto and co-workers in 
[127, 128, 129], by Falcon and Falcon in [130] and by Bonaldi et al [131] for actively 
cooled resonators used in a gravitational wave detector. A similar mapping was used 
by Berg to study the JR applied to gene expression dynamics [132]. 

5.2.3. Harmonic traps with changing stiffness Carberry ct al. [133] investigated the 
motion of a colloidal particle in a harmonic trap whose stiffness is suddenly changed 
from one value to another. They thus verified the TFT (97) for fit given by (98). 
For strongly localized initial conditions, this TFT has been verified experimentally in 
[134]. Gomez-Solano et al. inferred the fluctuations of the heat exchanged between a 
colloidal particle and an ageing gel which bears some similarity to a time-dependent 
stiffness [135]. 

5.2.4- Non-linear potentials Blickle et al. [136] measured the work distribution for a 
colloidal particle pushed periodically by a laser towards a repulsive substrate. This 
experimental set-up was the flrst one for colloidal particles that used effectively non- 
harmonic potentials. The pdf for work is distinctly non-Gaussian but still in good 
agreement with theoretical predictions based on solving the Fokker-Planck equation. 
This agreement justifles a posteriori the crucial assumption that the noise correlations 
are not affected by the time-dependent driving. Moreover, a DFT for p{w) (92) was 
checked for this periodic driving with a symmetric protocol. Sun determines p^w) for 
a potential that switches between a single well and a double well [137]. 

5.2.5. Stochastic resonance For a colloidal particle in a double well potential that 
is additionally subject to a modulated linear potential to generate conditions of 
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Figure 2. Distribution of entropy production p(As*°*) for a colloidal particle 
driven along a periodic potential for two different values of external force /[144]. 
The insets show the total potential v(x) — fx. The different histograms refer to 
different trajectory lengths. 

stochastic resonance [138], distributions for work, heat and entropy were measured 
and calculated in [139, 140]. Other numerical work using the concepts of stochastic 
thermodynamics to investigate stochastic resonance includes [141, 142, 143]. 

5.2.6. NESS in a periodic potential In an experiment for this paradigmatic geometry 
shown in Fig. 1 which correspond to the Langevin equation (1), the pdf for total 
entropy production has been measured and compared to theoretical predictions 
[144], see Fig 2. Characteristically, for short times, this pdf exhibits several peaks 
corresponding to the number of barriers the particle has surmounted. Examples for 
the pdf for entropy production have also been calculated in [145]. For long times, the 
asymptotic behaviour of this pdf in the form of a large deviation function has been 
calculated numerically in [146] where an interesting kink-like singularity was found. 
The same quantity has also been derived using a variational principle [147]. 

5.3. Underdamped motion 

5.3.1. Torsion pendulum as experimental realization A driven torsion pendulum 
differs fundamentally from colloidal particles since here the inertia term becomes 
accessible. In a series of experiments reviewed in [35], Ciliberto and co-workers have 
investigated the pdf for various quantities with the aim of checking which ones obey 
FT like relationships. 

In [148, 149], the JR and the Crooks relation were used to determine the "free 
energy difference" for the torsion pendulum for linear and periodic forcing. In [150], 
pdfs for the external work were determined for three different types of protocols for 
the time-dependent force in (38). (i) For a linearly ramped force, the pdf for starting 
in equilibrium was found to obey a TFT relation even for short times. Since a linear 
ramp corresponds to a time-dependent driving, one would, in fact, not expect a TFT. 
It is found here only because the work distribution for this linear system is Gaussian 
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and should obey the JR which constrains mean and variance such that the TFT is 
vahd. Alternatively, an explicit calculation of the pdf shows the same result, (ii) 
Starting in a quasi-steady state of the linear ramp, the pdf for 10"^^ no longer obeys 
the TFT for short times as expected, (iii) Likewise, for periodic driving, a DFT type 
for the work distribution is found only in the long time limit as expected. For the 
latter two cases, the finite time corrections have been calculated. 

The same group investigated the pdf for the heat for similar protocols [151]. 
In agreement with both the general theoretical expectations and their explicit 
calculations the pdf for heat does neither obey a TFT for short times nor even the 
DFT asymptotically for long times since the internal energy is not bounded for an 
harmonic oscillator. 

The pdf for changes in the stochastic entropy and the total one were reported for 
periodic driving of this system in [129]. Once the system has settled in the periodic 
steady state, for an integer multiple of the period the functional R becomes the total 
entropy change which thus fulfills a DFT as found experimentally. 

5.3.2. General theoretical results Using path integral techniques, Farago determined 
the statistics of the power injected by the thermal forces into an underdamped particle 
and found it to be independent of an underlying confining potential [152]. 

In a series of papers for a moving trap, Taniguchi and Cohen investigated pdfs 
for work and heat as well as various FTs using the path integral representation 
[153, 154, 155, 156]. They also point out the ambiguity (or freedom) to define time- 
reversal in this particular system. 

For both a moving and a breathing trap, Minh et al. calculated work weighted 
propagators for underdamped motion [157]. FTs for underdamped Brownian motion 
were studied by Lev and Kiselev by transforming from the momentum to the energy 
variable [158], by Fingcrlc for the relativistic version [159] and by Iso ct al. for 
motion near black holes [160]. Sabhapandit determined the work fluctuations of a 
randomly driven harmonic oscillator [161] which was studied experimentally in [162]. 
FTs for underdamped motion in the presence of two heat baths were investigated in 
[163, 164, 165]. 

5.4. Other sy. stems 

Ever since the DFT for entropy production has been formulated, there have been 
attempts to show whether it is fulfilled in a specific system both numerically and 
experimentally. Beyond the "clean" cases discussed above for which the dynamics of 
the relevant degrees of freedom is clearly compatible with a stochastic Markovian 
dynamics, there are a number of studies for other system where a theoretical 
understanding is more challenging. Such studies include an early theoretical work with 
three simple dissipative deterministic models [166], experimental [167, 168, 169, 170] 
and numerical [171, 172, 173, 174, 175, 176, 177, 178] work for granular matter or 
dense colloids, experimental work for turbulence [179, 180, 181], numerical work for 
a shell model in turbulence [182] and one on the role of hydrodynamic interactions 
[183], and experimental work for liquid crystals [184, 185], a vibrating plate [186], 
and self-propelled particles [187]. Characteristically for the experimental systems just 
mentioned, one cannot necessarily expect that these can be described by a stochastic 
Markovian dynamics for the relevant variables. Similarly, in the numerical works either 
the equations of motion are not of the Langevin type (or deterministic thermostatted 
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Figure 3. Network with five states comprising three cycles (1245), (234), and 
(12345) and the corresponding transition rates. Without the state 3, this network 
would be a unicyclic one. 

ones) or, if they are, not all variables are monitored which effectively amounts to some 
coarse-graining. Since for these cases, the FT's have not been proven, such tests give 
valuable hints on possible extensions beyond their established realm of validity. 

Two general aspects for putting results of such case studies in perspective are the 
following ones. First, the putative validity of a DPT for the quantity R is typically cast 
in the form of checking for a constant slope of the quantity limt_j.oo[ln[p(/9t)/p(— pt)]/t 
where pt is the time averaged rate corresponding to the quantity R = tpt for which 
we have formulated the FT. Since such a plot is necessarily asymmetric in pt [172], for 
a non-trivial statement the contribution of higher order terms like p^ must be shown 
to be negligible which requires a large enough range of studied pt-values. Moreover, a 
large enough t is necessary. Second, in bulk systems often only "local" quantities can 
be investigated which would require local forms of the FT's. From the perspective of 
a stochastic dynamics, this amounts to integrating out other slow degrees of freedom 
or some type of coarse-graining under which one can not expect the FT's to hold 
necessarily. We will come back to this issue at the very end of this review. 

6. Dynamics on a discrete set of states 

6.1. Master equation dynamics 

The derivation of the FTs in Sect. 4 is based on the behavior of the weight for 
a stochastic trajectory under time reversal or the other operations generating the 
conjugate dynamics. Therefore, they hold for any kind of stochastic dynamics, in 
particular, for a master equation type of dynamics [188, 189] on a discrete set of 
states {n}, see Fig. 3. 

Examples of such systems include random walks and, more generally, diffusive 
processes on a lattice, birth-death processes, growth processes on a lattice, 
conformational changes between discrete states of a biomolecule or chemical reaction 
networks. The latter two classes of systems differ from the previous ones since they 
typically occur in a well-defined thermal environment. This feature imposes additional 
constraints on the dynamics as discussed in Sect. 9 below. In this section, we focus 
on the stochastic dynamics with arbitrary transition rates. 

6.1.1. Transition rates and probability currents Transitions from a state to to a 
state n occur with a rate Wmn(A) which may be time-dependent according to the 
external protocol A(t). For case of notation, we will write u)m„(r) or u)m„(A) for the 
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more explicit i(j,„„(A(t)). In principle, one could distinguish different transitions or 
"channels" connecting the same two states from which we refrain here for notational 
simplicity. 

The probability p„(t) to find the system at time r in state n evolves according 
to the master equation 

drPnir) = ^ [Wmn(r)Pm(T) - W™n(r)p„(T)] (110) 

given an initial distribution Pn{0). To each link (mn) one can associate a (directed) 
probability current 

jmnir) = P,niT)Wmn{T) - P„ (T)ti;„,„ (t) = -j„„i(T). (Ill) 

6.1.2. Two classes of steady states For time-independent A, the system eventually 
reaches a steady state provided the network is "ergodic", i.e., any two states are 
connected through a series of links as we will always assume in the following. The 
time-independent stationary probabilities can be written as 

K(A) = exp[-0„(A)] (112) 

which defines the analogue of the non-equilibrium potential. For small networks, there 
is an elegant graphical method to determine pfj from given rates [189, 190]. 

Steady states fall into two classes depending on whether or not the detailed 
balance condition (DBC), 

p^(A)w„„(A) =p^,(A)u;„„(A), (113) 

is fulfilled. The first case corresponds to genuine equilibrium, the second one to a 
NESS. In the latter case, there are non- vanishing steady state probability currents 

Jmn = Pm^mn ~ P„UJnrn — ~Jnm- (H^) 

Since knowing is not sufficient to distinguish a genuine equilibrium from a 
NESS, Zia and Schmittmann [191, 192] suggested to characterize a NESS by its 
stationary distribution and its stationary currents j^„. Then the same NESS 
could be generated by a whole equivalence class of possible rates Wmn since any two 
sets of rates with 

P'mi'U^m.n - w'„^„) = p'„{Wnm. ~ wl^m) (US) 

would lead to the same NESS, i.e., the same stationary distribution and currents. In 
the following, we will adapt the view that a system is characterized by a definite set 
of rates WmnW and a protocol A(t) from which all other quantities can, in principle, 
be derived. II 

A "distance" from equilibrium quantifying the amount of violation of the DBC 
has been introduced in [198]. A Lyapunov function for relaxation towards a NESS is 
discussed in [199]. Master equation dynamics as a gauge theory is formulated in [200]. 
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Figure 4. Trajectory n(T) of total length t jumping at discrete times {tj} 
between states. 



6.1.3. Path weight and dynamical action Wc first characterize the fluctuating 
trajectories. A trajectory nir) with < r < t starts at ng and jumps at times 
Tj from nj to ending up at state rit after J jumps, see Fig. 4. Defining for each 
state the instantaneous total exit rate 



the conditional weight for a trajectory exhibiting no jump at all is given by 
p[n(r) = nol^io] = cxp[- / dr r„o(T)]. 

The weight for a trajectory with J > 1 jumps at times {tj} is given by 
p[?i(t)|71o] = exp[- I dr r„o(T)] x 

dr r„+ (r) 



(116) 



(117) 



J r 



(118) 



with Tj+i = t. Averages with these weights will be denoted by (...) in the following. 
In analogy with the continuous case (5), these expressions define an "action" 

dr r„„ (r) 



^[ri(T)] = — lnp[n{T)\no] 
J 



E 



dr r„+ (r) 



(119) 



().^. Entropy production 

6.2.1. Stochastic entropy The concept of stochastic entropy can be transferred 
immediately from the Langevin case to the discrete one as [20] 

s(T) = -lnp„(,)(T). (120) 

II For completeness, we mention a complementary approach where rates are derived by imposing 
mean currents as constraints [193, 194, 195, 196]. For a relation to the minimum entropy production 
principle, sec [197] and references therein. 
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It is obtained by first solving the master equation (110) for p„(t) with a given initial 
distribution and then plugging into it the specific trajectory n(r) taken from this 
ensemble. 

The equation of motion for stochastic entropy becomes 



Sir -Tj)\n—^——. (121) 



ri(T) , Pnj-{Tj) 



The first term shows that even if the system remains in the same state, stochastic 
entropy will change whenever the ensemble is time-dependent either due to a non- 
equilibrated initial state or due to time-dependent rates. The second term shows the 
contributions from each transition. The change in stochastic entropy during time t is 
given by 



As = / dTs{T) = - \npnM + lnp„o(0)- 
Jo 



(122) 



6.2.2. Time reversal Time- reversal as a choice for the conjugate dynamics works 
analogously to the Langevin case with n'l'(r) = n{t — r) and At(r) = \{t — r). Using 
the weights (117,118), it is trivial to check that 

There are two justifications for identifying Ri with the entropy change of the 
surrounding medium As™. The first is the analogy to the Langevin case, where this 
term turned out to be the dissipated heat (divided by temperature). In the absence 
of a first law for the master equation dynamics, which would require further physical 
input not available at this general stage, this identification is by analogy only. Second, 
it turns out that the sum of the system entropy change as defined in (122) and the so 
identified medium entropy, 

As*°' = As + As™, (124) 

will have all the properties required from a total entropy production. From the 
expression (123) an instantaneous entropy production rate in the medium can be 
identified as 

W - + (tj ) 

that makes the contributions from the individual jumps obvious. Combining this 
relation with (121), one obtains 



Atot/ N drPnir) 



(r) = - 



Pnir) 



p -(rj-)^«„7„+(^i) 
^ Sir r,)\n—^-^-^^^. (126) 



6.2.3. Ensemble level By averaging these trajectory-dependent contributions for 
entropy production, one obtains expressions derived much earlier within the ensemble 
approach [23, 189, 190, 201]. On a technical level, using 

i^^ir - Tj)dn.-,,.+ {T) \ = ^Pm{T)w,nn{T)d^n{T) (127) 
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valid for any set of quantities {dmnli one obtains 



Prn{T)Wmn{T) 

ran 

and 



r>tot/ \ / -tots t \ (• \ 1 PrnVJ^mnyT) /i ■-lo^ 

S [T) = {s ) = 2^p™(r)'u;™„(r)ln — j—, (128) 



S^{t) = (s'") = Vp„(r)u;™„(r)ln^^^^4 (129) 

mn 

which should be compared with (35) and (36), respectively. 

6.2.4- Splitting entropy production Following the Langevin case, we can split the 
entropy production in the medium (123) into two contributions 



As'" = As^^ + As'''' (130) 



with 



where Xj = A(tj) and 



As'''^ = yin - , ' \, , (131) 



characterizing the entropy change associated with maintaining the corresponding 
steady state and the one associated with time-dependent driving, respectively. It 
is simple to rewrite As'^'^ as the discretized version of — + d\(^ as in (23). This 
excess entropy production has a nice geometrical interpretation along a path in the 
parameter space analogous to the Berry phase in quantum mechanics [202]. 

A somewhat different splitting of total entropy production on the trajectory level 
was introduced in [101, 102, 103] writing 

As'°' ^ As''^ + As"'' (133) 

with the adiabatic entropy change As^'^ = As'^*^ and the non-adiabatic one As"^ = 
As"'^ + As. 



6.2.5. Dual dynamics The dual dynamics is defined by rates 

wl^iX) s ii;„™(A)K,(A)/p^(A). (134) 

These rates lead to the same stationary state as the original dynamics, = p^. 
However, the stationary currents are reversed according to 

jK^nW = -JnrnW- (135) 

In complete analogy to the Langevin case, by comparing the weights for the original 
with the dual and the dual-reversed dynamics, one obtains 

i?i = As'''' and i?i = As'^'', (136) 

respectively. 
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6.3. FTs for entropy production and "work" 

6.3.1. General validity With these identifications, all FTs from Sects. 3.3 and 4 
involving the various forms of entropy changes apply under exactly the same conditions 
as stated there provided the occasional x (and XQ,Xt) is trivially replaced by n (and 
nQ,nt). 

The only variable not defined yet is the analogue of work. For networks that at 
constant A fulfill the DBC (113), one can identify a (dimensionless) internal energy 
as 0„(A) = — Inpfj(A) that plays the role of the potential V{x, X)/T in the Langcvin 
case with a free energy identical to zero. At this stage, there is indeed no point 
in identifying a non-trivial A-depcndcnt free energy. Consequently, work along a 
trajectory corresponds to dissipated work and can be identified in analogy to (15) 
with 



With this identification of work, the FTs from Sects. 3.2 and 4 involving work hold 
for this master equation dynamics as well (setting there T = 1). It should be stressed, 
however, that without a more physical microscopic understanding of the network, this 
concept of work (and heat, if one wanted to promote As™ to this status) is a purely 
formal one without real physical meaning. 

For networks that do not fulfill the DBC (113), there is no unique way of assigning 
internal energy to a state without further physical input, and, hence, no sensible way 
of identifying work even formally. Naively keeping (137), as sometimes suggested [203], 
fails as the counter-example of a discretizcd version of the driven overdampcd motion 
on a flat ring easily shows, since ~ const implies w ~ 0. 

The statistics of rare events contributing to these FTs can also be studied through 
a "mapping" of the master equation to a Schrodinger equation and then analyzing the 
corresponding path integral [204, 205]. Finally, a somewhat formal general IFT was 
derived in [206]. 

6.3.2. Experimental case studies Experimental work measuring the distributions of 
these quantities with the perspective of "testing" the FTs is yet scarce. The arguably 
simplest non-trivial network is a two-state system with time-dependent rates. Any 
such two-state system necessarily obeys the DB condition. Such a two-state system 
was realized experimentally by driving an optical defect center in diamond with 
two lasers. The distributions for work (137) and for the entropy change of system, 
medium and total, were measured and compared to the theoretical predictions [88, 90]. 
Characteristically, for comparably short times these distributions show quite intricate, 
distinctively non-Gaussian, features, see Fig. 5. 

6.3.3. Analytical and numerical case studies Entropy production and the FT for the 
simplest discrete system which is in essence a random walk biased in one direction by 
applying an external field was studied in the context of a ratchet model in [207], for 
a rotary motor in [208, 209] and for transport through a membrane channel in [210]. 
Simple three and four state systems were investigated in [211, 212]. The statistics of 
dissipated heat for a driven two-level system modelling single electron transport has 
been calculated in [213]. 

Entropy production on a lattice model both for a simple reaction-diffusion scheme 
and for transport was investigated in [214] with an attempt to clarify the occurence 




(137) 



CONTENTS 



42 



- a) 




2 


1)) 








1-0 

1 










O-'i 


■J 
















-0.6 -0.3 0-3 0-6 -1-5 -1 -0-5 0-5 1 1-5 
R R 



Figure 5. Probability distribution for the "work" (137), denoted here R, in a 
driven two-level system for two different lengths of trajectories. The histogram 
are experimental data, the full curve a theoretical calculation, for more details, 
see [88]. 



of a kink in the rate function at zero entropy production. In another variant of 
the reaction-diffusion scheme, the violation of an FT caused by breaking microscopic 
reversibihty in the sense that some backward transitions are forbidden were studied 
in [215, 216]. Entropy production for a model of cyclic population dynamics was 
investigated in [217] and for effusion of a relativistic ideal gas in [218]. 

The analogue of work distributions for a spin system in time-dependent magnetic 
fields was investigated in [219, 220] and for time-dependent coupling constants in 
[221]. The surface tension in the three-dimensional Ising model is determined through 
simulations using the analogy of the JR in [222]. 

The interplay between a non-equilibrium phase transition and singularities in the 
entropy production has been investigated for a majority vote model [223], for driven 
lattice gases [224, 225] , for wetting [226] and for kinetically constrained lattice models 
[227]. 



6.4. FT for currents 

For a network in a NESS, currents obey an FT as first derived by Gaspard and 
Andrieux [228, 229, 230, 231] exploiting the decomposition of such a network in cycles 
as introduced by Schnakenberg [189] and using concepts from large deviation theory 
[232] . For a concise derivation of the current FDT using the formalism developed in 
Sect. 4, we write the total entropy production along a trajectory in the form 

As*°* = UaASa + ASr- (138) 

a 

Here, ria is the number of times a cycle a has been completed in clockwise (ria > 0) 
or anti-clockwise {ria < 0) direction during this trajectory leading to a fluctuating 
current Ja = ria/t for each cycle, see Fig. 3 for an example of cycles in a network. 
The entropy production associated with each cycle 

ASa^ ^ In^ (139) 

is also called the affinity of this cycle. The remainder Asr collects the contributions 
arising from those parts of the trajectory that do not contribute to a full cycle. Clearly, 
the current Ja is uneven under time-reversal and hence qualifies as a possible variable 
Sa with ea ~ —I for the general SSFT (93) which thus becomes 

Pii-Ja}) 



P{{ja}) 



= exp[-tJ2^SaJa] (e-^-'1{Ja}). (MO) 
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For large t, As,,, and hence the second factor remains of order 1. It can thus be ignored 
when taking the logarithm in the long-time limit leading to the current FT 

lim \ In 4^^^ = E (^^l) 

t^oo t p{{'Ja}) ^ 

For a network coupled to different reservoirs at different temperatures or chemical 
potentials, the cycle affinites AS'^ arise from externally imposed afhnites J^j, as 
discussed in more detail in Sect. 10.2 below. These affinites give rise to mesoscopic 
currents 

Ju=Y.^-< (142) 

a 

where c?^ are a generalized distance counting how much each cycle contributes to the 
respective current. Expressed in these currents, the FT reads 

1 , P({-7fc}) T ^ fTA-i\ 

hm -In— f T\=/^^kJk- (143) 

In this derivation, it is crucial that all currents that contribute to the entropy 
production (either on the cycle or on the mesoscopic level) are included. 

Generalizations of such an FT have been derived for just one current in [233], for 
networks with multiple transitions between states in [234], to a current not related 
to entropy production for a growth model on a lattice in [235] and for networks 
with a particular topology in [236]. Geometrical and topologial aspects were studied 
in [237, 238, 239, 240]. Periodically driven systems were investigated in [241] and 
a relation to supersymmetry is made in [242]. In another extension, Hurtado et 
al. have derived an "isometric fluctuation relation" that compares pdfs for currents 
with different orientations [243]. 

The FT for entropy production has been discussed for various chemical reaction 
networks in the papers by Gaspard and Andrieux quoted above, and, more recently, 
be applied to transport in mesoscopic devices which, despite their quantum character, 
can often still be described by a master equation whenever coherences can be, or are, 
ignored. For a few recent examples, see, e.g., [204, 205, 244, 245, 246, 247, 248, 249, 
250, 251, 252, 253]. 

There is a large literature arising from the recent progress of understanding 
current fluctuations in NESS in general, not necessarily related to the FT, for which 
the review by Dcrrida could serve as a point of departure [254]. 



7. Optimization, irreversibility, information and feedback 

7.1. Optimal protocols 

7.1.1. General aspects The integral fluctuation theorems like the Jarzynski relation 
hold for any external protocol A(r) and any time interval t. An optimal protocol A*(r) 
is the one that extremizes the mean of a functional of the trajectory like work or heat 
for given initial value Xi = A(0) and final value A/ = X{t) of a control parameter and 
a fixed total time t allocated to this process. 

Mean work as objective functions is arguably the most relevant case. For t — >■ cx), 
the minimal mean work required for a transition is given by the free energy difference 
AJ" = J-{Xf) — J-{Xi). For any finite time t, the mean work should be larger and the 
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question for the optimal protocol becomes non-trivial. Understanding this problem 
will allow to extract the maximum amount of work from a given free energy difference 
in finite time. 

Formulated as a variational problem, the optimal protocol obeys a quite 
complicated Euler-Lagrange equation which is non-local in time since changing 
the control parameter at time ti affects the work increment at all later times t2. 
Crucial insight into general features of the solution, however, has been obtained by 
investigating case studies involving harmonic potentials [255]. As a general feature, 
jumps of the optimal protocol were found that are absent in a linear response treatment 
[256]. 

A second motivation for minimizing the work could be an attempt to improve 
convergence of the Jarzynski estimate to obtain free energy differences since one might 
expect that a small mean work may also to lead to a smaller variance. Due to the 
non-linearity of exp[— w/T], however, one should rather find the optimal protocol for 
minimizing (cxp[— 2w/T]} which turns out to have jumps as well [257]. 

7.1.2. Overdamped dynamics The generic jumps in the optimal protocol were first 
found in case studies involving harmonic potentials [255]. The simplest case is a 
process where the center of a harmonic potential V{x,X) = (x — A)^/2 is shifted from 
Ai = to Ay in a finite time t. Such a shift does not involve any free energy difference. 
Hence, the mean work required for this task will approach for t — > oo. For a finite 
time, the optimal protocol can be calculated analytically by expressing the mean work 
as a functional of the mean position of the particle which renders the problem local 
in time. The optimal protocol (in dimensionless units for time) 

A(r)=A/(r + l)/(t + 2) (144) 

involves two jumps 

AA = A(0+) - Ao = A/ - A(t-) = A//(t + 2) (145) 

at the beginning and the end of the process. The physical reason for, e.g., the first 
jump is the fact that with this jump the dissipation rate is constant throughout the 
process. If the trap moved with constant speed without initial jump, the friction 
would slowly build up at the beginning of the process which ultimately would imply 
stronger dissipation. The size of the in this case symmetric jumps at beginning and 
end vanishes as < — > 0. 

Similar jumps have also been found in a second case study where the stiffness of 
a harmonic potential V{X,x) = Ax^/2 is changed in finite time from an initial value 
Ai to Ay [255] . For overdamped motion of a dipol in a magnetic field that switches the 
orientation, the optimal protocol can even show a degeneracy [258] . Further examples 
for optimal protocols involving non-linear potentials were studied numerically in [257]. 

An intriguing mapping of this optimization problem to deterministic optimal 
transport like mass transport by a Burgers velocity field has been discussed in 
[259, 260]. For total entropy production as objective function, turning an earlier 
scaling argument [261] into a mathematical proof, a general bound can thus be 
derived, AS"*"' > C/t valid for any t, where C depends on the given initial and final 
distribution [262] . The key point is that this optimization takes place in the space of 
all probability distributions rather than in a restricted space of driving potentials with 
a few variational parameters. In the latter case, the ^ 1/t behavior will hold only for 
long times. 



CONTENTS 



45 



7.1.3. Underdamped dynamics For under-damped dynamics, the optimal protocol 
involves even stronger singularities at beginning and end of the process given by 
additional 5-peaks in the protocol [263]. Physically, these terms guarantee that the 
particle at the beginning acquires, and at the end looses, a finite mean momentum 
instantaneously which again minimizes total dissipation. 

7.1.4. Discrete dynamics Optimal protocols can also be investigated for master 
equation dynamics on a discrete set of states. A simple case has been studied as 
a model for a quantum dot with a single energy level E connected to a reservoir with 
chemical potential 1^1 = E — e(r). The optimal protocol for an externally controllable 
gap eir) and given e(0) and e(t) minimizing the mean work W = J^^ dTp{T)e{T), where 
p{t) is the probability that the energy level is occupied, shows jumps in the optimal 
e*(T) at beginning and end which are nicely explained in physical terms in [264]. A 
more general approach to the optimal protocol connecting arbitrary given initial and 
final distributions is given in [265] . 

7.2. Quantifying time's arrow 

The concepts developed for deriving the FTs can also lead to a more quantitative 
understanding of irreversibility. In Sect. 4, the time-reversed process was introduced 
as a mere tool for deriving the FTs. By considering such a time-reversed process 
seriously and comparing it to the original time-forward process, one can indeed derive 
an interesting relation between dissipation and irreversibility. An essential tool in this 
analysis is the relative entropy or Kullback-Leibler distance 



between two distributions p{y) and q(jj). Essentially, this quantity measures how 
distinct the two distributions are [266]. 

We present here the stochastic version of the basic idea first introduced using 
Hamiltonian dynamics [267, 268]. For a process with a time-dependent potential 
y(x. A) and its time-reversal, which start and end in the respective equilibrium states, 
the quantity R in the generalized FT (79) becomes R = w — AJ^. This FT thus implies 



where the average is conditioned on the value Sa of an arbitrary functional S'c([x(t)] 
along the trajectory with a definite parity under time-reversal. By choosing 
Sa = w — AT, averaging the logarithm of (147) yields 



This relation, which can be seen as a consequence of the CFT (61), shows that the 
dissipated work determines how different the distributions for this quantity along 
the forward and the backward paths are. Likewise, a large difference of these two 
distribution implies a substantial dissipated work. 

By choosing Sa = x(ti) = {t — ti), i.e., the state of the system at any 
intermediate time ti, one gets from (147) the lower bound 




(146) 



(exp[-(w - AF)]|Sq) ^ p\eaSa)/p{sa) 



(147) 



(w) - AF=D[p{iu - AT)\\p\-{w - AT) 



(148) 



{w) ~ AT > D[p{x{t,))\\p\x\t,))]. 



(149) 
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on the dissipated work. In contrast to this stoehastic ease, for a Hamihonian dynamics, 
one obtains an equahty in (149) due to Liouvihe's theorem [267]. For both types 
of dynamics, further coarse graining, i.e., fooking at the distributions for a variable 
y ~ i/(x(ti)), leads to a lower bound on the dissipated work since relative entropy 
decreases under coarse graining [266] as nicely illustrated in the present context in 
[269]. 

Similarly, by choosing ti =^ t, one immediately obtains the inequality 

{w) - AT > D[p{^m\p'''i{^m (150) 

which bounds dissipation by the distinguishability of the instantaneous distribution 
with the corresponding equilibrium distribution as derived and discussed in [270]. 

It is trivial to derive similar relations for processes involving genuine steady states 
that at constant control parameter reach a NESS rather than equilibrium as pointed 
out in [271]. Essentially, in (147-150), one has to replace (w) - AT by (As"' + A0) 
where (f> is the non-equilibrium potential (11). 

Related inequalities have been discussed for transitions between specified initial 
and final states [272]. Relations between other information theoretic measures and 
non-linear averages of work and entropy along non-equilibrium trajectories have been 
derived in [273, 274, 275]. An intriguing relation between generating information 
and dissipation has been made for DNA replication in [276] with a corresponding 
pedagogical introduction in [277]. 

7. 3. Measurement and feedback 

7.3.1. Feedback and the second law According to the Kelvin-Planck formulation of 
the second law, one cannot extract work from an equilibrated system at constant 
temperature without leaving any traces of this process somewhere else. The situation 
becomes apparently different if information about the state of the system during 
this process becomes available through a measurement as the classical example of 
Maxwell's demon and the Szilard engine reviewed in [278, 279] demonstrate. 5] Based 
on the result of a measurement, one can choose a particular protocol for a control 
parameter which will indeed allow either to extract work in a cyclic process or, in a 
non-cyclic process, to extract more work than the corresponding free energy difference 
of initial and final equilibrium state. These statements are still compatible with the 
second law since erasing the information acquired will also cost free energy according 
to Landauer's principle. Taking this additional effect into account, the ordinary 
second law is restored. Typically, for discussing these processes within stochastic 
thermodynamics the cost of measurement and erasure process is first ignored in the 
problem of how to convert the acquired information into work (most efficiently) as it 
also will be ignored in the following discussion of the main concept where we use an 
approach based on FTs. Related earlier work will be briefly mentioned in Sect. 7.3.5. 

7.3.2. Measurement and information For a quantitative description, we assume a 
system evolving according to a master equation as introduced in Sect. 6. If the 
system at time ti is in state ni = n(ti), a measurement yields a result j/i with the 
probability p{yi\ni) = p{ni\yi)p{yi) /p{ni,ti). Here, p{n,t) is the ordinary solution 
of the master equation for the given initial condition and p{yi) the probability for 

^ For an instructive criticism of one of the assumptions of the Szilard engine, see [280]. 
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obtaining the result yi irrespective of ni. The (trajectory-dependent) information 
acquired in this measurement is [281] 

i{ni,yi) =hi[p{ni\yi)/p{ni,ti)] = \n[p{yi\ni)/p{yi)]. (151) 

Upon averaging with p(yi|ni), this trajectory dependent information becomes the 
relative entropy D[p(ni\yi)\\p(ni)] which still depends on the result yi of the 
measurement. Further averaging over yi leads to the mutual information 

dyMyi)D[p{ni\yi)\\p{ni)] (152) 
dyi / dnip{ni,yi)i{ni,yi). (153) 



7.3.3. Sagawa-Ueda equality (SUE) and a generalization After a measurement, the 
control parameter A(r, yi) for the subsequent evolution ti < t < t depends uniquely 
on the outcome yi leading to the probability distribution pi(n,r|?/i). For a system 
with a time-dependent potential V{n,X), i.e., a system that at any fixed A reaches 
a genuine equilibrium state, Sagawa and Ueda have generalized the JR (58) to this 
feedback-driven process in the form [281, 282] 

(exp[-(w- AJ" + i)]) = 1 (154) 

which implies for the maximal mean extractable work = — (w) the bound 

VF°"'<-AJ- + Z (155) 

with I = (i). Thus, acquiring information through a measurement allows to extract 
more work than what one would get from a process without feedback. 

The original formulation of the SUE requires the notion of a free energy difference 
for initial and final state. For transitions involving genuine non-equilibrium states, i.e., 
those that at constant control parameter reach a NESS, the analogous relation 

(exp[-(As*°* +i)]) = 1 (156) 

with the inequality 

(As*°*) > -I (157) 

hold true as well.+ 

A concise proof of (156), which will be valid with a minor modification for (154) 
as well*, not requiring explicit time- reversal follows from exploiting the IFTs (84,85) 
[283]. By using As*°* = As + As'°, splitting the last term into the two contributions 
associated with the two time intervals i = {0 < t < ti) and ii = (ti < t <t), making 
the total entropy change of the system explicit with As = — lnp(nt, t\y) -|-lnpo("-Oi 0)j 
and the specific expression for the information (151), the Ihs of (156) can be written 

+ Note that even for detailed balanced systems, the equalities (154) and (156) are different since, in 
general w — AT ^ As*°*. 

* For proving the SUE (154), one only needs to replace Pi(»it, t|j/i) by p'^'^(^^t, t|j/i) in (158). 
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as 

{—, — re — — re "pi{nt,t\yi) ) 

= ( — -!— e"'^'*'"|ni = TOi) p(ni,ii) X 

X (e-^^>?pi(nt,t|2/i)|ni\ =1. (158) 
\ la 

Introducing conditioned averages on the two intervals i and ii eliminates the explicit 

factor l/pi(rii,ii|j/i). The underlined term is 1 for any m\ due to (84). Likewise, 

the subsequent summation over m\ is 1 due to (85). The SUE thus holds force for 

trajectory-averages still conditioned on the results y\. Of course, further averaging 

over all possible outcomes y\ is allowed. This proof (as the original one) is easily 

extended to multiple measurements. Thus, the Sagawa-Ueda equality and its variant 

(156) with the corresponding inequalities hold for any number of measurements 

[282, 283, 284, 285, 286]. 

For processes involving genuine non-equilibrium states, the generalization of the 

Hatano-Sasa relation (66) to processes with feedback in the form [283] 

(exp[-(As*°' - i^s^^ -h i)]) = 1 (159) 

with the inequality 

(As'°*} > (As^"^) - 1 (160) 

follows as easily starting with conditioned variants of (87). The bound (160) is much 
stronger than (157) since (As^"^) will typically scale with the total time t. For systems 
that at constant A exhibit detailed balance, As''*^ = 0, in which case (159, 160) become 
(156, 157). 



1.3.Jf.. Efficiency of Brownian information machines For a cyclically operating 
information machine, where measurements are repeated at regular intervals separated 
by tm [287], the inequality (157) implies that one can extract at most a mean power 
bounded by 

PI^™* < i, (161) 

where X is the rate with which information is acquired. Likewise, for processes 
involving transitions between genuine non-equilibrium states, the inequality (160) 
implies 

W"""^ <W"' - {q^^)+i. (162) 

If the rate of acquiring information is large enough, i.e., if I > ((j'^'^), the extracted 
power can exceed the power VF'" required to sustain these non-equilibrium steady 
states as demonstrated explicitly with a simple example in [283]. Characteristically, 
the power extracted from such a machine becomes larger, the smaller the intervals tm 
between the measurements are. 

A quite natural definition for the efficiency [287, 288] of such a Brownian 
information machines obeying < < 1 is in the first case 

ri = W^™7i (163) 

and, analogously, 

r) = W°'''l[W"' - {q^'^)+i\ (164) 

for the second case. 
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7.3.5. Further theoretical work and case studies Several theoretical studies have 
investigated various aspects of such feedback driven processes for stochastic dynamics. 
Kim and Qian have considered an underdamped particle controlled by a velocity- 
dependent force [289, 290]. This problem has been analyzed from the perspective 
of total entropy production in [291]. Suzuki and Fujitani investigate Brownian 
motion both under a time-dependent force [292] and for linear systems more generally 
[293]. Similarly, Sagawa and Ueda illustrated their concept using a particle that 
is transported in a movable harmonic trap and can still extract work from the 
surrounding heat bath [281]. Feedback driven transport for ratchet- type systems has 
been optimized in [294, 295, 296]. Maximum power for such a model has been studied 
in [297]. Information-theoretic and thermodynamic concepts have been combined in 
[298, 288, 299]. The thermodynamic cost of a measurement has been modelled in 
[300]. 

The optimal protocol for extracting the maximal work from cyclic processes for 
particles in harmonic traps with adjustable center and stiffness based on imperfect 
positional measurements has been calculated in [301] where it was shown that the 
bound (155) cannot be saturated if only the center of the trap is under control. Only 
by additionally adjusting the stiffness, all the information can be recovered provided 
an infinite time is allocated to the process. For such a machine, the efficiency at 
finite cycle time has been calculated in [287]. The issue of saturating this bound has 
been investigated in more depth introducing the notion of "reversible" feedback by 
Horowitz and Parrondo [302, 303]. A model for the cost of erasing information using 
a Brownian particle in a double well potential was discussed in [304] . 

1.3.6. Experimental illustrations Experimentally, the Sagawa- Ueda equality has 
been demonstrated by using an ingenious set-up involving electric fields that upon 
measuring the position of a colloidal particle on a "stair" prevent that the particle 
slides down a step that it has just climbed by thermal excitation [305]. In another 
experiment, Landauer's principle has been illustrated using a colloidal particle trapped 
in a modulated double-well potential. The mean dissipated heat indeed saturates at 
the Landauer bound in the limit of long erasure cycles [306] . 

7.3.7. Hamiltonian dynamics for microcanonical initial conditions Deviating from 
the restriction to stochastic dynamics as applied generally in this review, I mention a 
few recent studies that use Hamiltonian dynamics and feedback since they provide an 
additional perspective on what has just been described. The Kelvin-Planck statement 
of the second law does not hold for micro-canonical initial conditions which indeed 
allow to extract work, i.e., to decrease the mean energy from a Hamiltonian system 
by manipulating an external control parameter [307]. Specific examples have been 
given for a harmonic oscillator in [308], for a particle between movable walls [309] 
and for motion in a double well potential [310]. While for such microcanonical initial 
conditions no measurement is necessary, these results could be applied to an initially 
canonical ensemble if the energy of the system is measured with subsequent adaption 
of the protocol of the control parameter in a feedback process. As shown explicitly in 
[310], the full analysis including the cost of erasing information exorcizes this "demon" 
and restores the ordinary second law. 
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8. Fluctuation-dissipation theorem (FDT) in a NESS 

8.1. Overview 

8.1.1. FDT in equilibrium Equilibrium systems react to small perturbations in a 
quite predictable way formally expressed by the FDT, see, e.g., [311]. The response 
of an observable A at time T2 to a perturbation h applied at time ti can be written 
in the form of an equilibrium correlation function as 

T5{A{T2))/5h{T^)\h=o - Ti?^'(T2 - n) - drM{r2)B{T,)f^, (165) 
where the conjugate variable 

B = -dhE (166) 

follows from the energy E{K) of the system. Here it is assumed that for any small 
fixed h the energy of the system is still well-defined. This FDT is the formalization 
and generalization of Onsager's regression hypothesis that states that the decay of an 
excitation is independent of whether it has been generated externally by a force (or 
field) /i or by a thermal fluctuation. This theorem is of great practical significance 
since it allows to predict the response to a perturbation without ever applying one 
just by sampling the corresponding equilibrium fluctuations either in experiments or 
in simulations. Characteristically, the same B holds for any A and any time difference 

T2 - Ti . 

8.1.2. FDT in a NESS Whether a similarly universal relation exists for NESSs 
has been addressed using various approaches since the seventies. For an underlying 
stochastic dynamics, Agarwal has expressed the response function by a correlation 
function involving the typically unknown stationary distribution [312]. Bochkov and 
Kuzovlev [12, 313, 314] and Hanggi and Thomas [315] have derived a variety of 
formal expressions for stochastic processes. A comprehensive review of the general 
relation between fluctuations and response including, in particular, deterministic 
chaotic systems is given in [316]. 

More recently, taking up a theme introduced earlier [317], Harada and Sasa 
derived a relation where the "violation" of the equilibrium FDT in a NESS was related 
to the rate of energy dissipation for a Langevin system [318, 319] later generalized to 
a description in terms of a density fleld [320] . For the special case of a driven colloidal 
particle it was shown in [321] that the FDT in the NESS could be obtained from the 
equilibrium FDT (165) by subtracting from the right-hand side a second correlation 
function involving the local mean velocity. This result suggested that in the locally 
co-moving frame the Onsager hypothesis could be restored which was later extended 
to sheared systems [322] and proven for general diffusive dynamics in [323, 324]. Thus, 
for these systems, the decay of an excitation around a local NESS is still the same 
whether generated externally or by the thermal fluctuations still present in the NESS. 

A concise formal derivation and discussion of the general FDT in a NESS has 
been given by Baiesi et al. [325, 326, 327]. The response of particular observables was 
treated at the same time by Frost et al. [328]. In [329], it was then shown that the 
latter result holds indeed for any observable and that the FDT for a NESS becomes 
particularly transparent when using the concept of stochastic entropy with its splitting 
into a total and a medium one. In this latter work, the apparent multitude of FDTs 
in a NESS was rationalized in terms of an equivalence relation holding for observables 
in NESS correlation functions. 
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An elegant synthesis using mathematically somewhat more demanding concepts 
has just been given in [330]. An extension of these concepts to obtain an FDT around 
non-stationary non-equilibrium states is derived in [331]. A connection with gauge 
fields is made in the geometrical approach of [332] . 

8.1.3. Effective temperature The derivation of recent exact versions of the FDT for 
a NESS which as a result typically express the response function by a sum of two 
correlation functions should be distinguished from the phenomenological concept of 
an effective temperature that has been reviewed in [333] . Originally introduced in the 
context of ageing systems, it can be formulated also for a NESS. Simply stated, guided 
by the equilibrium form (165) an effective temperature is defined as 

T^*f(A, T2 - ri) = d,, {A{t.2)B{t^)YIRa{t2 ~ n) (167) 

where Ra is now taken in the NESS. In general, T°^ will depend on both the 
observable A and the time- difference T2 — ti which upon Fourier transformation 
corresponds to frequency u. Obviously, this concept can become meaningful only 
if these dependencies are not too pronounced. 

From a theoretical point of view, a strictly observable and frequency- independent 
T'^^ follows for a Langevin system like (42) with f = if the non-equilibrium conditions 
are caused by an additional "active" noise t] with correlations {T]{T2)ri{Ti)) = 2{T°^ — 
T))i6(t2 — Ti). For a linear system and active noise correlated on a scale r^^ T<=*^ wih 
depend on frequency. For lot^'^ ^ 1, one gets the ordinary temperature, whereas for 
uiT^^ ^ 1 the enhanced fluctuations lead to a larger T°^ . 

On the other hand, if the non-equilibrium is generated by a non-conservative force 
of field, such a simple reasoning is no longer possibe. Still, in interacting systems one 
often finds numerically good agreement with the concept of an effective temperature 
as briefly mentioned in Sect. 8.4 below for sheared suspensions. A fundamental 
understanding when and why this is the case in general seems still to be missing. 

For insight into the frequency and observable dependence for specific models and 
systems, see, e.g, [334, 335] for a binary Lennard- Jones mixture in a simple shear 
flow, [336] for a glassy model system, [337, 338, 339, 340] for simple interacting model 
systems, [341, 342, 343] for simple Langevin systems, and [344] for field-theoretical 
models. 

Examples of investigating biophysical systems using an effective temperature 
include [345] for hair bundle oscillations, [346, 347] for the cytoskeleton, [348, 349] 
for filament oscillations in an active medium, [350] for self-propelled particles, [351] 
for vesicle and [352, 353] for red-blood cell fluctuations. If the response of such a 
system acts effectively "against" the perturbation, the effective temperature becomes 
negativ as occasionally found in these studies. Such an observation shows that this 
concept should not be taken too literally. 

An intriguing, apparently simple system in this context is "hot Brownian 
motion" where a diffusing particle heated by a laser acts as a local heat source 
[354, 355, 356, 357] for which in simulations various "effective temperatures" were 
determined [358]. Local heating in the context of the FT has been discussed 
theoretically in [359]. 
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8.2. Derivation and discussion 

In this section, we sketch the derivation of the various forms of the FDT in a NESS 
from a unifying perspective for a general Markovian dynamics on a discrete set of 
states. Since overdamped Langevin systems can always be discretized, this case is a 
very general one. We follow the concepts introduced in [325, 329] which were briefly 
reviewed in their continuum version in [360]. Earlier related work for a Markovian 
dynamics on a discrete set of states making somewhat more explicit assumptions on 
observable and rates for spin models include [361, 362, 363, 364, 365, 366, 367, 368] 
and for ageing in supercooled liquids [369]. 



8.2.1. Equivalent correlation Junctions in a NESS Wc consider a class of NESSs 
with rates Wmn{h) that depend on a perturbation h. The stationary distribution of 
the master equation dynamics (110) obeys 

^irn„(/i)K(/l) =0 (168) 
n 

with the generator 

{h) ~ 5mn^Wrnk{h). (169) 
k 

For fixed /i, any dynamic information is fully contained in the propagator 

Gki{T) = prob[n(r) = k\n{Q) = /] (170) 
for which the master equation (110) implies the evolution 

drGkl ='^^LkmGml{'r) ='^^Gkm{'r)Lmi. (171) 

m m 

In a NESS, denoted in the following with (...)'*, two point correlation functions for 
state variables of the form A{t) ~ J2m ^■mSn{T)m arc given by 

{A{T2)B{n)r = ^ A™G™„(t2 - ri)B„K, (172) 

mn 

if T2 > Ti. Using (169) and (171), a time derivative with respect to the earlier time 
can thus be written as an ordinary two-point correlation function 

drAA{T2)B{n)r = {A{T2)C{T,)r (173) 

with 

m ra 

= Y^{Bn - B^)p'^^W,nn/Pn- (174) 
m 

Besides state variables as obscrvablcs we will also need current variables of the 

type 

i?(r)^^,5(r-r,)d„,-„,+ (175) 
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that yield (i„^. -„^.+ whenever a corresponding transition takes place. Their NESS 
average is given by (£'(t))^ = J2mnPm'^rnndmn- If D{t) shows up in a correlation 
function at the earlier time ti, wc get 

{AiT2)D{Ti)y = Am.GmliT2 " Ti)plwkldkl (176) 

mkl 

= {A{t2)E{t,)Y (177) 

with 

En = ^PlWkndkn/Pn- (178) 
k 

These relations imply, in particular, that the formal current variable 

i3(r)^^J(r-t,)(B„^+-B„^-) (179) 
i 

obeys 

{A{T2)B{Ti)r = dr, {A{t2)B{ti)Y (180) 

which demonstrates, quite expectedly, that time derivatives can be pulled in and out 
a NESS correlation function straigthforwardly. 

The fact that NESS correlation functions can have the same value if the variable 
at the earlier time is written differently gives rise to an equivalence relation denoted 

by 

c^O(2)(^) (181) 

if 

{A{T2)O^^Hn)Y = {A{t2)O^^Ht^)Y (182) 

holds for all A and times n < T2 [329] . For the variables defined above we obviously 
have 

D{t) = E{t) and B{t) = C(r) (183) 

which summarizes how current variables and time derivatives can be replaced by state 
variables in NESS correlation functions. This freedom will explain why apparently so 
differently looking FDTs can be derived for a NESS. 

8.2.2. Equivalent forms of the FDT The apparent plethora of FDTs can be 
rationalized by starting with an expression for the response function using the path 
weight (119). In the presence of a time-dependent perturbation ft.(r), the mean value 
of the observable A{t) is given by [325, 329] 

(A(r)) = ^A(r)p[n(r);/.(r)|no]< 

n(r) 

X. Mn{T);h(T)\nQ\ r , ^| -, ^ ^ ' 

^) p[n(r)|no] 

The response function 



Ra{t2 ri) ^ 5{A{T2))/5h{T^)\^^, {A{t2) {r,)Y (185) 
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can be expressed by a two-point correlation function in the unperturbed NESS by 
evaluating (184) with the action (119) as 

B^in) = 5A[n{r)- h{r)] / 5h{T,X^, (186) 

= - ^w,,(T)kan(T)k + ^S{t-Tj)a^-^+. (187) 
k j 

where 

amn = dh\riWjnnih)\h^Q. (188) 

This form of the conjugate variable (with the superscript^ alluding to the derivation 
through the path weight) is convenient since it allows to determine the response 
function by measuring a correlation function that requires only knowledge about how 
the rates depend on the control parameter which is easily availabc in simulations. The 
more formal aspect that the first term in (187) arises from the time-symmetric part of 
the action and the second one from its time-asymmetric one is emphasized and further 
exploited in [325, 326, 327]. 

A second equivalent representation of the conjugate variable is obtained by 
replacing (by following the scheme (175-178)) the second (current) part in by 
its equivalent state variable form which leads to B^ = with 

Bn= ^^kCtnk + WknaknPl/Pn (189) 

k k 

= Y.^hL-^^^)\h=^PklPn- (190) 
fe 

The last equality follows from expanding (169) in h and the definition (188). This 
expression for the conjugate variable involving only state variables can also be derived 
by straightforward time-dependent perturbation theory of the Fokker-Planck equation 
as originally derived by Agarwal (hence, the superscipt") [312]. Using this expression, 
however, requires knowledge of the stationary distribution which for interacting 
systems with many degrees of freedom is not easily available in either simulations 
or experiments. 

Finally, as a third, arguably physically most transparent form of the conjugate 
variable, it is easy to check explicitly that —dhS = i?" by expanding (168) in h and 
following the recipe of how to pull a time-derivative into a correlation function given 
in the previous subsection. Consequently, one has [329] 

Ra{t2-ti)= -{dhs{Ti)Y (191) 

= {dhs''\n)Y - {dhs'°\n)r. (192) 

The first form expresses the response function as a time-derivative of a correlation 
function where the observable A{t2) is correlated with the /i-derivative of the stochastic 
entropy at ti. This form of the conjugate variable is actually unique if one wants to 
write the response function as a time derivative of a correlation function. Moreover, it 
allows a physically transparent interpretation by splitting it into the sum of medium 
and total entropy production as shown in the second line. 

8.2.3. Comparison with equilibrium FDT For a comparison with the equilibrium 
FDT assume that the steady state is a genuine equilibrium state for h = 0. In fact, 
two classes of such systems should be distinguished. 
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First, if the system is not only in equilibrium at /i = but also at small /i, the 
stationary distribution is given by the Boltzmann distribution 

Pl^{h) = exp{-K(/i) - F{h)]/T}, (193) 

where Enih) is the internal energy and J-{h) the ^.-dependent free energy of the 
system. The stochastic entropy obeys s„(/i) = — h\p'^{h) ~ [En{h) — J'(/i)]/T. Along 
an individual trajectory, J-{h) is constant and hence we have 

TdhSnih)^,^^ = dMh)^„^o (194) 

Inserting this equivalence into (191), the FDT acquires its well-known equilibrium 
form (165,166) involving the observable conjugated to h with respect to energy. 

Second, a system may be in equilibrium at /i = but driven into a NESS even at 
constant small h. The paradigmatic example is a perturbation through shear flow for 
which there is no corresponding E(h) for any h ^ 0. For such systems, the equilibrium 
FDT can still be written in the form (191) but also in the pure state form with B'^ 
from (189,190). 



8.2.4- Systems with local detailed balance A further comparison between the 
equilibrium and the NESS-FDT is instructive for systems for which the perturbation 
enters the ratio of the rates in the form of a local detailed balance condition 

W,nn{h) U^m»(0) r, , 

77-: = 7^exp[/id„„/T], (195) 

where d„in = —dnm is the distance conjugate to the field covered by the transition 
m ^ n. For ft, = 0, the system is supposed to be in genuine equilibrium with averages 
denoted by {■■.)°'^; for h = ho ^ a genuine NESS denoted by (...)" is reached. 
In equilibrium, using the global detailed balance condition (113), one easily verifies 
9hS™ = and hence one has the equilibrium FDT 

Ti?^^(T2 - Ti) = {A{T2)B-{n)r = {Air2)dhs"'iTi)r\ (196) 

On the other hand, the NESS-FDT in the form (192) always holds. Since for such 
systems 

= 5] <5(t - T,)d„^.-,„^. + /T (197) 
j 

is independent of h, the recipe for getting the FDT in a NESS from the equilibrium 
FDT is to keep as a first term the observables showing up in the correlation function 
but to evaluate the latter under NESS conditions and to subtract an expression 
involving the total entropy production [329] . 



8.2.5. Generalized Green-Kubo relations In equilibrium, the Green-Kubo relations 
express transport coefficients like conductivity or viscosity by time-integrals over 
equilibrium correlation functions of the corresponding currents. Based on the FDT 
derived above, it is possible to derive similar relations between transport coefficients 
in a NESS and appropriate current-current correlation functions [370] as illustrated 
for a simple models of molecular motors in [371]. This approach of studying the linear 
response of a NESS should be distinguished from extensions of the Onsager symmetry 
relations to the non- linear response coefficients of an equilibrium system [372] . 
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8.3. Colloidal particle on a ring as paradigm 

The overdamped particle driven along a periodic potential, see Fig. 1, as discussed 
in Sect. 2.2 can serve as paradigm for illustrating the different versions of the FDT 
[329]. 

8.3.1. Equivalent correlation functions The equivalence relation introduced in Sect. 
8.2.1 for variables occuring in correlation functions in a NESS exists for continuous 
variables as well. For a discrctizcd position variable, jump rates can easily be derived 
from discretizing the path integral. Going then through the steps as in Sect. 8.2.2 
shows that the equivalence 

x'^2v''{x) ~ liF{x) (198) 

can be used in a NESS correlation function at the earlier time.jj The mean local 
velocity v'^{x) has been introduced in (12). Sometimes, the generalization 

g{x)x ^ g{x)[2v\x) - nF{x)] - fiTd,g{x) (199) 

is useful which can be derived similarly.jf 

8.3.2. Three equivalent forms First, consider a NESS generated by a force /o which 
is further perturbed by an additional delta-like force impuls acting at time ri. The 
response function can be written as correlation function in the three equivalent forms 

Ti?A(T2 - ri)|;,„^o = {A{t2)[x - iiF{x)\rj2Y (200) 
= {A{T2W{x)-piF(x)]\,,Y (201) 
= {A{t2)[x~v^{x)]\,,Y. (202) 

The first form follows from applying perturbation theory to the path integral 
expression. The advantage of this expression is that it does not require explicit 
knowledge of the stationary distribution. By replacing the velocity with the 
corresponding state variable as shown in (198) one obtains the second form. This 
expression can also easily directly obtained from perturbation theory of the Fokker- 
Plank equation as in the original derivation [312]. Finally, a simple linear combination 
of the first two lines leads to the third form originally first derived in [321]. In this form, 
both the additive correction to the equilibrium form TRa{t2 — n) = {A{t2)x{ti))°'^ 
and the significance of a locally co-moving frame become apparent. 

8.3.3. Generalized Einstein relation The Einstein relation (3) connecting the bare 
mobility fj, embedded in a viscous fluid with its diffusion constant D has arguably been 
the first form of an FDT which was based on a microscopic understanding of thermal 
motion. This relation has many manifestations in more complex soft matter systems 
as reviewed in [48]. If such a particle is in a periodic potential V{x), the diffusion 
coefficient 

D[V{x)] = lim [{x^{t)) - {x{t)Y]/2t (203) 

tl In equilibrium, this equivalence becomes x = fj,dj:V{x) whore the crucial sign difference compared 
to naively ignoring the noise in the Langevin equation (1) should be noted. 

ft Applied to a NESS correlation function with A(t2) = 1, this relation leads to {g{x)x)'' = 
{g{x)[2v''{x) - iiFix)])" - fiTid^gix))" = {g{x)v'' [x))" which corresponds to (34) applied to a NESS. 
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and the effective mobility 



/i[y(x)] = a^(i)i/=o 



(204) 



still obey D[y(a;)] = Tfi\y{x)\ even though both terms are exponentially suppressed 
if the barriers exceed the thermal energy. 

In the presence of a non-zero base force, an effective diffusion constant D\V{x), f] 
and a mobility fi[V{x),f] as in (203) and (204) evaluated at a finite force, respectively, 
are still defined. The effective mobility is the time-integrated response function. 
Hence, the generalized Einstein relation between D[V{x),f] and iJ,[V{x),f] follows 
from integrating (202) from T2 = —oo to T2 = ti as [321] 



T^^[V{x)J]^D[V{x)J]+ dT[{x{T) - {±)][v%x{0)) - {x)r] (205) 



which shows how the "violation" of the usual Einstein relation can be expressed as 
an integral over velocity correlation functions. This relation is a simple example of a 
Green-Kubo relation generalized to a NESS [370]. Its form for other simple geometries 
like two-dimensional motion in the presence of a magnetic field has been studied in 
[373] and for a discrete model showing anomalous diffusion in [374]. 

8.3.4- Experiments The generalized Einstein relation (205) has been measured 
experimentally in [375]. Significantly, in this experiment, the extra integral term 
in (205) can be about four times as big as T^[V{x),f] which shows clearly that 
this experiment probes a genuine NESS far from any linear response regime of an 
equilibrium system. Still, the description of the colloidal motion by a Markovian 
Brownian motion with unaltered thermal noise and a drift obviously remains a faithful 
representation. The very fact that around a critical force / ~ inax\dxV{x)\ the 
diffusion coefficient becomes quite large is known as giant diffusion [376, 377]. 

The time-resolved version of this FDT has been studied experimentally in [378] 
where it was shown that even though the different correlation functions (200-202) are 
theoretically equivalent their statistics can be vastly different. Not surprisingly, the 
variant (201) involving only state functions shows better convergence properties than 
the ones requiring x. The response not to a force but to a change in the amplitude of 
the periodic potential was studied experimentally in [379, 380]. 

8.4. Sheared suspensions 

For studying the relation between fluctuations and response in interacting non- 
equilibrium systems, a colloidal suspension in shear flow provides a paradigmatic case. 
Such a system follows a dynamics as introduced in Sect. 2.6.3 with u(r) — jye^ (or 
the corresponding underdamped version) and some pair interaction V. 

One obvious question is to investigate the generalized Einstein relation between 
the self-diffusion coefficient Dij (7) and the mobility /i^ j (7) of a tagged particle which 
both become tensorial quantities in such an anisotropic system. Szamel [381] studied 
these quantitities analytically using the memory-function formalism. Kriiger and 
Fuchs [382] have studied this relation analytically and numerically near the glass 
transition. Our numerical study in the fluid phase [383] revealed that for moderate 
densitities the results can surpringiy well be expressed as an effective temperature since 
the ratios Dii{'j) / fiul/f) of the diagonal elements become isotropic with a roughly 
quadratic increase with shear rate. This effective temperature which turns out to 




CONTENTS 



58 



be the kinetic one can be rationalized by comparing this interacting system to a 
harmonically bound single particle in shear flow [384] . The response to a perturbation 
in the shear rate has been investigated in [322] and the one to a static external long 
wave- length perturbation in [385, 386]. 

Further studies of the general FDT for sheared suspensions include the integration 
through transient formalism [387, 388]. One advantage of this approach is that all 
quantities can be expressed in terms of (albeit complicated) equilibrium correlation 
functions. The response to a time-dependent additional shear strain has been studied 
numerically in [322] using essentially the form (191) that makes the excess compared 
to the equilibrium case explicit. The relation between the violation of the equilibrium 
FDT and energy dissipation using field variables has been addressed in [320]. 

9. Biomolecular systems 

9.1. Overview 

Single molecules and (small) biomolecular networks constitute a paradigmatic class 
of systems to which the concepts of stochastic thermodynamics can be applied. 
Conformational changes of single molecules have become observable through a variety 
of methods often summarized as single molecule techniques [389, 390, 391]. There 
are essentially two ways of exposing such a molecule that is embedded in an 
aqueous solution of well-defined temperature containing different solutes at specified 
concentrations to non-equilibrium conditions. First, one can apply a (possibly time- 
dependent) mechanical force if one end is connected via polymeric spacers to the tip 
of an AFM or to beads in an optical tweezer. This set-up allows to study, e.g., force- 
induced unfolding of proteins. Another source of non-equilibrium are unbalanced 
chemical reactions catalyzed by the enzyme under study. In combination with a 
mechanical force, this set-up allows in particular to resolve individual steps of a 
molecular motor and to measure force-velocity curves of such a molecular machine 
[392, 393]. 

The fluctuating conformations of biomolecules in non-equilibrium can be 
described in two ways [34, 394, 395]. First, one can model the observable degrees 
of freedom like the end to end-distance of a protein by a continuous degree of 
freedom subject to a Langevin equation. Such an approach is particularly appropriate 
for studying force-induced un- and refolding of biopolymers, in particular, with 
the perspective of recovering free energy differences and even landscapes from non- 
equilibrium experiments as reviewed in Sect. 9.3 below. 

Second, one can identify discrete, distinguishable states between which (sudden) 
transitions take place as it has often been done to model molecular motors [396, 
397, 398, 399, 400, 401, 402, 403, 404, 405, 406, 407, 408, 409, 410, 411, 412]. In 
most of these works the focus has been on elucidating the cycles involved in the 
action of the motor and on deriving force-velocity curves and their dependence on 
ATP and ADP concentrations. A combination of both types of models has been 
used for describing molecular motors by a Langevin dynamics in a ratchet potential 
that depends explicitly on the current chemical state of the motor as reviewed in 
[413, 414, 415, 416]. 

From the perspective of stochastic thermodynamics, one would like to formulate a 
first law, discuss entropy production and derive the corresponding fluctuation theorems 
on the single molecule level. Within a discrete state description, a first law along 
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an individual trajectory has been discussed for single enzymes in [84, 208, 417], for 
molecular motors in [400, 404, 405, 406, 407, 408, 409], and for small biochemical 
reaction networks in [228, 418, 419]. Fluctuation theorems without explicit reference 
to a first law were discussed for such systems in [401, 410, 420, 421, 422, 423, 424]. 

From a theoretical point of view, there are essentially two new aspects that enter 
the stochastic thermodynamics of biomolecular systems beyond a naive combination 
of the stochastic thermodynamics of colloids as developed in Sect. 2 and the discrete 
dynamics as introduced in Sect. 6. First, the rates are not arbitrary as in Sect. 6 but 
are rather constrained by thermodynamic consistency [189, 190, 425] as discussed 
in detail below. Second, each of the states visited along a stochastic trajectory 
contains many microstates. Transitions between these (unobserved) microstates are 
fast so that thermal equilibrium is reached within each state. Transitions between 
the states, however, are slower, observable and can be driven by external forces, flows 
or chemical gradients. As a consequence, each of the states described by stochastic 
thermodynamics carries an intrinsic entropy arising from the coupling to the fast 
polymeric degrees of freedom and to those of the heat bath. This effect must be taken 
into account in any consistent identification of heat on the single trajectory level 
[84, 408, 426, 427]. Some of the earlier studies quoted above missed this contribution 
and, hence, failed to identify the dissipated heat correctly. 

How these systems can be described from the perspective of a thermodynamic 
engine will be pursued in Sect. 10. 

9.2. Biopolymers with continuous degrees of freedom 

In this section, we show how a continuum description fits into the framework developed 
for systems without relevant internal degrees of freedom like a colloidal particle. This 
systematic presentation is largely original, even though some of the essential concepts 
are implicit in most of the works dealing with free energy recovery discussed in Sect. 
9.3 below. 

9.2.1. Thermodynamic states from a microscopic model A biopolymer contains a 
large number of coupled microscopic degrees of freedom most of which will not 
be accessible in experiments. Still, these microscopic degrees of freedom affect 
processes on a larger scale that are described by stochastic thermodynamics. The 
microscopic configurational degrees of freedom collectively denoted by ^ are subject to 
a microscopic potential energy <&(^, A) containing the interactions within the molecule 
(and possibly with some of the surrounding solvent and solute molecules). The 
dependence on A allows for an external potential like an AFM or an optical tweezer 
whose position can be controlled through A. 

Under non-equilibrium conditions, an external force (or field or fiow) is applied 
to the molecule leading to conformational changes apparent through, e.g., a changing 
end-to-end distance. Such a quantity is an example of a meso-scale description that 
involves a certain number of variables denoted by x. Each such state effectively 
comprises many micro states. Formally, one can split all microstates {^} in classes Cx 
such that each ^ belongs to exactly one Cx. The dynamics of x is supposed to be slow 
and observable whereas equilibration among the micro states making up one state x 
is fast. Under this crucial assumption, the conditioned probability p(^|x, A) that a 
microstate is occupied is given by 

p(Clx, A) = exp[-($(e. A) - F(x, A))/r] (206) 
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with the constrained free energy 

F(x, A) EE £;(x, A) - T5(x, A) = -Tin ^ exp[-$(e, A)/r], (207) 

the constrained intrinsic entropy 

5(x, A) = -aTF(x, A) = - ^ p(^|x, A) lnp(C|x, A) (208) 

and constrained internal energy 

i?(x,A) = ^ $(e,A)p(e|x,A). (209) 

This model includes but is more general than a more conventional description of 
the configurational potential in the additive form 

$(x, e, A) = $"(x, A) + $'"*(x, A) + $"^^<i(e, A) (210) 

made up by a system, a coupling (of arbitrary stength) and a potential for the degrees 
of freedom of the medium, which all may depend on the control parameter. Here, all 
degrees of freedom are split into those of the system x and those of the heat bath 
^. By replacing $(^, A) in (206,207) with $(x, ^, A) and summing without constraint 
over all ^, the relations (206-209) remain true for the potential (210). 

9.2.2. First law For a time-dependent A(t), representing, e.g., the center of a moving 
laser trap, the increment in applied work reads 

dw=Y. 9A$(e,A)p(e|x,A) d\ (211) 

= 9a-F(x, \)d\ = dF{x, A) - VF(x, A)dx. (212) 

In the first equality, for a changing external parameter the work arising from 
the microscopic interaction $(x. A) is expressed as an average over all microstates 
contributing to the state with (fixed) x. The second equality follows with (206). 
Compared to the expression in the colloidal case (15), the essential difference here 
is that F(x,X) is a free energy rather than a bare potential, i.e., internal energy. 
Consequently, the first law that, of course, should involve internal energies rather 
than free energies becomes 

dw dE{x, A) +dq = dF{ii, A) + rd5(x. A) +dq. (213) 

This relation together with (212) implies for the increment in heat 

dq = -VF{x,X)dx-TdS{:>c,X), (214) 

which makes the contribution to heat that arises from the intrinsic entropy S'(x, A) 
clear. 

For a practical evaluation of the work, one would have to know _F(x, A), which, 
in general, has a complicated A-dependence if the microscopic potential $(^, A) is 
genuinely A-dependent. However, if the external potential couples only to the slow 
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Figure 6. Schematic view of a protein stretched by a bead in a laser trap. 
In a meso-scale description, the configuration is characterized by the positions 
X = {xi , a;2, 2:3, X4, X5 , x)} where x is the position of the bead. The control 
parameter A denotes the center of the trap. 



degrees of freedom x as typically assumed, see Fig. 6, a significant simplification 
occurs. In this case, one can write 

F(x, A) = ^^°(x) + F(x, A) = i;°(x) - r5"(x) + l/(x, A), (215) 

where the quantities with superscript*^ are the thermodynamic potentials (207-209) of 
the molecule for constrained slow variables x in the absence of the external potential. 
As a consequence 

dw ^ dxV{^,\)d\ (216) 

which becomes trivial for the typical case of a harmonic potential T^(x, A) — k{xi — 
A)^/2, with Xi the relevant coordinate for the coupling and k the effective stiffness of 
the AFM tip or optical trap centered at A(r). 

9.2.3. Dynamics For the dynamics of the slow degrees of freedom one has the 
Langevin equation 

x = ^[-VF(x,A)]+C (217) 
with the noise correlations as in (43). Likewise, the Fokker-Planck equation reads 

5,p(x, r) = V(^Vi^(x, A)p(x, r) + T^Vp(x, r)), (218) 

Compared to the discussion in Sect. 2.6.2 the key point here is that whenever 
states carry intrinsic entropy, the gradient of the free energy (rather than of internal 
energy) has to show up in the Langevin and Fokkcr-Planck equations since for any 
fixed A, the system has to reach equilibrium with the Boltzmann factor 

p<='i(x, A) = cxp[-(F(x, A) - ^(A)/T] (219) 



with the A-depcndent free energy 

T[X) = -Tin / dxexp[-F(x, A)/r]. 



(220) 
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9.2.4- Entropy production The stochastic entropy along the trajectory x(r) becomes 

s(T) = -lnp(x(r),r) (221) 

where p(x, r) follows from solving the Fokker-Planck equation (218) with an 
appropriate initial condition. For such a system with intrinsic entropy, the total 
entropy production along a trajectory during time t 

As'°'ee f dTs*°' = / dT[s{T)+S{x,X)+q/T] (222) 
Jo Jo 

contains three contributions rather than two as in the case without relevant intrinsic 
degrees of freedom. 

9.2.5. FTs In principle, the FTs holds true in the presence of intrinsic entropy as 
well provided the latter is taken into account properly. The crucial point is that the 
master functional Ri defined in (71) when using as conjugate process the time-reversed 
one becomes 

Ri^ As"'' +q/T (223) 

where 

As'"' = 5(xt,At)-^(xo,Ao) (224) 

is the change in intrinsic entropy along the forward trajectory. This result follows 
from evaluating the action in the path weight corresponding to the Langevin equation 
(217) and by using the first law (213) integrated along the trajectory. 

As a consequence, the FTs involving entropy production discussed in Sects. 3.3 
and 4 essentially hold true with the replacement 

As'"^ As'" + As'"*. (225) 

All FTs involving total entropy production hold true unmodified. 

The FTs involving work as defined in (212) hold true as well. In particular, the 
JR stands with AJ" = J'{Xt) — J'i^o) where the free energies have been defined in 
(220). The reason why the intrinsic entropy does not spoil these relations is the fact 
that both the work and the force in the Langevin equation are determined by the free 
energy F(x, A) very much in the same way as the bare protential V{x, A) determines 
the corresponding quantities in the colloidal case. Loosely speaking, the results of the 
simpler case hold true provided one replaces the potential V{x, X) by the free energy 
(i.e, potential of mean force) F{x, A). 

In the presence of intrinsic entropy, the FDT (192) must be modified accordingly 
by replacing by s™ + s'"' . 

9.3. Free energy recovery from non- equilibrium data 

9.3.1. Hummer-Szaho relation (HSR) and variants From a practical perspective, 
arguably the most relevant FT for biomolecules is the Hummer-Szabo relation 
[10, 428]. As a kind of JR resolved along a reaction coordinate, it allows to determine 
the free energy landscape F°(x) from non-equilibrium work measurements through an 
external potential y(x. A) conditioned on a fixed value of x. It reads 

cxp[-i^°(x)/T] = exp[(l^(x,AO - J-(Ao))/T] 

X (exp[-i(;/T],5(xt - x)). (226) 
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The right hand side is evaluated by measuring the aceumulated work w as a function 
of position irrespective of the particular t. 

A concise derivation [21] of the HSR starts with the IFT (80). With the necessary 
replacement (225), the initial equilibrium distribution po(x, Aq) ~ exp[— (_F(xo, Aq) — 
J-{\q)/T]^ the free choice pi(xj) = 5(xt — x), the first law (213) and the assumption 
(215), it follows within a couple of lines. 

A variant of the HSR not requiring the position histograms in (226) can be derived 
for a harmonic coupling F(x, At) [429]. Moreover, similarly as the CFT generalizes 
the JR by including information from the time-reversed process, bidirectional variants 
of the HSR have been derived and tested in model calculations [430, 431]. 

9.3.2. Experiments The first experimental application of the JR to biomolecules was 
the determination of the free energy involved in partially unfolding RNA hairpins 
[432]. The CFT was first applied in another experiment measuring the free energies 
in RNA hairpins and RNA three-helix junctions [433]. In a series of experiments, 
Ritort and co-workers have used the CFT to determine the free energy involved in 
unfolding DNA hairpins [87, 434, 435, 436]. The free energy involved in unfolding the 
multidomain protein, titin, has been measured in [437] using a simplified variant of 
the JR leading to some criticism [438, 439]. The CFT has been used to determine free 
energy changes induced by mechanically unfolding coiled-coil structures [440, 441] and 
different topological variants of a protein [442] . Axis-dependent anisotropy in protein 
unfolding was investigated using the HSR in [443] . The free energy landscape derived 
from the HSR has been compared to equilibrium measurements in [444] 

9.3.3. Numerical work As relatively scarce real experimental studies using the FTs 
still are, as large is the number of "numerical" experiments illustrating the use of 
the HSR and its variants for recovering free energy landscapes. The following brief 
list is necessarily incomplete. Model calculations for a single coordinate deal with 
the advantage of applying a periodic force protocol [445], with random forcing [446], 
comparison with "inherent structures" [447] , with motion in a periodic potential [448] , 
and with recovering an unknown spatially dependent mobility [449] . Multidimensional 
landscapes were reconstructed in [450]. Monte-Carlo or molecular dynamic simulations 
were used for an off-lattice model protein [451], for a protein domain in [452] and for 
a membrane protein in [453]. 

An important line of research in this context is to find methods for dealing with 
the error caused by having only finite (and even noisy) data for evaluating the non- 
linear averages involved in the JR and the HSR. Some of the papers dealing with this 
issue are [86, 137, 454, 455, 456, 457, 458, 459, 460, 461, 462, 463, 464, 465, 466, 467, 
468, 469, 470, 471, 472, 473, 474, 475, 476, 477, 478, 479, 480]. 

9.4. Enzymes and molecular motors with discrete states 

In this section, we show how the general framework for a Markovian dynamics on 
a discrete set of states can be adapted to describe the stochastic thermodynamics 
of enzymes and molecular motors in a thermal environment starting again from a 
microscopic model. Apart from keeping track of the intrinsic entropy of the states the 
essential point is to incorporate the enzymatic reactions involving solute molecules 
consistently. 
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9.4-1- Thermodynamic states The enzyme is in an aqueous solution which consists 
of molecules of type i with concentrations {ci} and chemical potentials {^i} enclosed 
in a volume y at a temperature T. It exhibits a set of states such that equilibration 
among microstates corresponding to the same state is fast whereas transitions between 
these states are assumed to be slower and observable. Under these conditions, one 
can assign to each state n of the enzyme a free energy F™^, an internal energy E'^"^, 
and an intrinsic entropy S"™^. As explicitly discussed in [427], these quantities follow 
from any microscopic model that specifies the energy of the microstates of enzyme 
and solution. They obey the usual thermodynamic relation 

despite the fact that the enzyme is small. Moreover, there is no need to assume 
that the interaction between enzyme and solution is somehow weak. In general, these 
thermodynamic variables of the enzyme depend on the concentrations of the various 
solutes. 



9.4-2. First law In this section, we discuss the first law for the three classes of (i) 
pure conformational changes, (ii) enzymatic reactions including binding and release 
of solutes, and (iii) motor proteins. 

(i) Conformational changes: If the enzyme jumps from state m to state n, the 
change in internal energy 

A£;™^ = s™" - £:™" = -q (228) 

must be identified with an amount q of heat being released into (or, if negative, being 
taken up from) the surrounding heat bath since there is no external work involved. 

(ii) Enzymatic reactions: The more interesting case are enzymatic transitions that 
involve binding of solute molecules Ai, their transformation while bound, and finally 
their release from the enzyme. Quite generally, one considers transitions written as 

n;+^rfA.^n+ + ^sfA, (229) 

i i 

where 1 < p < Np labels the possible transitions. Here, n~ and denote the 
states of the enzyme before and after the reaction, respectively. For = 0, this 
scheme describes pure binding of solutes, and for r^ — release of bound solutes. A 
transformation (like bound ATP to bound ADP + P^) can also be described by the 
above scheme with ''f = = and the understanding that the enzyme states contain 
the bound solutes, see [427] for a more detailed discussion. 
The free energy difference involved in such a transition 

AFp = AEp - TASp = AF;"" + AF;°' (230) 

has two contributions where 



^ pcnz ^ ^^cnz _ T/S^gcn^ ^ ^cnz _ ^onz (231) 
P P P rip 7ip ^ ' 

denotes the free energy change of the enzyme and 

AFf = AEf - TASf EE ^(,sf - rf )m, ^ A^^p (232) 

i 
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denotes the free energy change attributed to the solution in this reaction. Both free 
energy contributions can as usually be split into internal energy and intrinsic entropy. 

As in the case of pure conformational changes, one assigns a first law type energy 
balance to each reaction of type p (229). Once an initial state is prepared, in the 
closed system (enzyme plus solution) there is obviously no source of external work. 
Neither does the system perform any work. Hence, the heat released in this transition 
is given by minus the change of internal energy of the combined system [427] 

qp = -AEp = -AE;''^ - App - TASf. (233) 

This relation shows that the enzyme and the solution are treated on the same footing 
since only their combined change in internal energy enters. Since the heat is released 
into the solution acting as a thermal bath, the configurational change of the enzyme 
as well as binding and releasing solute molecules contribute to the same bath. 

(Hi) Molecular motors: Essentially the same formalism applies to an enzyme 
acting as a molecular motor often described by such discrete states. Most generally, 
if the motor undergoes a forward transition of type p as in (229) it may advance a 
distance dp in the direction of the applied force / (or, if / < 0, opposite to it). The 
special cases dp = (pure chemical step) or = = (pure mechanical step) are 
allowed. For dp ^ 0, the mechanical work 

^--h ^ (234) 

is applied to (or, if negative, delivered by) the motor. 

The motor operates in an environment where the concentration of molecules like 
ATP, ADP or Pj are essentially fixed. The first law for a single transition of type p 
becomes [427] 

qp = u-^-^i^ - AEp = fdp - AE;^^ - Afip - TASf. (235) 



9.4-3. Role of chemiostats: Genuine NESS conditions The more recent form (235) 
of the first law differs from an expression discussed previously for molecular motors 
[400, 404, 405, 406, 407, 408]. There, in the present notation and sign convention, the 
first law for a step like in (229) reads 

qp - - Ai?;"^ - A^p. (236) 

The difference between the two expressions for the heat 

qp-qp^ TASf (237) 

involves the entropy change in the solution resulting from the reaction. 

The physical origin of the two different forms arises from the fact that in the 
older work the enzyme is thought to be coupled to "chemiostats" providing and 
accepting molecules at an energetic cost (or benefit) given by their chemical potential. 
Introducing the notion of a "chemical work" 

^chem ^ (238) 

the first law is then written in the form 

Wp"''^" + wf"'" = A£;;"" + qp. (239) 
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The concept of chcmiostats is supposed to guarantee that the concentration of 
solute molecules remains strictly constant. Physically, it could be implemented by an 
ATP buffer of ATP-regeneration scheme that involves additional enzymes. From the 
perspective of stochastic thermodynamics as long as one focusses on single tranistions, 
however, it would be more appropriate to treat these additional enzymes and the 
chemical reactions they catalyze in the same way as the reaction involving the motor 
protein. It turns out that if these additional reactions operate quasistatically, then 
Op is the dissipated heat that under steady state conditions would enter an ensemble 
average [427]. Therefore, choosing the heat qp is appropriate whenever one deals with 
strict NESS conditions while not wanting to consider the heat involved in enforcing 
these conditions explicitly as an extra contribution. On the trajectory level for a single 
motor protein, there seems to be no sensible way for assigning qp instead of qp to an 
individual transition. 



9.4-4- Stochastic trajectory and ensemble A trajectory of the enzyme can be 
characterized by the sequence of jump times {tj } and the sequence of reactions {Pj^} 
where pj denotes the corresponding reactions (229) and aj = ± characterizes the 
direction in which the reaction takes place. 

An ensemble is defined by specifying (i) the initial probability Pn{0) for finding 
the enzyme in state n and (ii) the set of rates with which the reactions (229) take 
place in either direction. Both inputs will then determine the probability p„ (t) to find 
the enzyme in state n at time r. 



9.4.5. Rates and local detailed balance An identification of entropy production along 
the trajectory requires some input from the rates determining the transitions. For the 
simple case of pure conformational changes, m ^ choosing rates that obey 

^ = exp[-(F,r - FZ')/T] (240) 



is required by thermodynamic consistency. Indeed, only this choice guarantees that 
irrespectively of the initial conditions the ensemble will eventually reach thermal 
equilibrium, 

with the free energy of the enzyme 

jronz ^ _rin^ cxp[-i^,^"VT]. (242) 

n 

Fixing the ratio of the rates still leaves one free parameter per pair of states which can 
only be determined from knowing the dynamics of the underlying more microscopic 
model. 

The corresponding relation for transitions that involve enzymatic reactions (229), 
^ = exp[-AF,/r] = cxp[-(AF;- + A^ip)/T], (243) 

Wp 

and for transitions of molecular motors, 

= exp[-(AF;- + App - ^«7^'^)/T], (244) 

Wo 
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are somewhat less obvious. Essentially, three types of justifications for choosing such 
ratios can be given. 

First, even though microreversibility is often invoked it seem unclear how to 
obtain these ratios rigorously using this concept if chemical reactions arc involved. 

Second, one can derive (243) using the following argument. For any enzymatic 
reaction there will be concentrations {c^^} of the solutes such that the enzyme will 
reach equilibrium. For these particular equilibrium concentrations, a choice of rates 
respecting (243) is mandatory as in the case of pure conformational changes. If one 
now assumes that (i) the reaction rates obey the mass action law and that (ii) the 
concentrations and the chemical potentials are related by the ideal solution expression, 
;xi(ci) = + T\n{ci/c^), then the form (243) follows. 

Third, more recently it has been shown that by requiring a consistent stochastic 
thermodynamic description on the trajectory level, one can indeed derive these 
conditions on the rates using rather mild assumptions [427]. 

In all cases by invoking the respective first laws (228,233,235), the ratio of the 
rates can also be written in the form 

^ = exp[A5p + qp/T] (245) 

Wp 

showing that this ratio is determined by the change of intrinsic and medium entropy 
involved in this transition. This important relation should be compared with (20) 
in the colloidal case where the continuum version of such a ratio involves only the 
dissipated heat since there is no relevant intrinsic entropy change given here by AS p. 
Similarly, for a biopolymer within a continuum description, the relation (223) shows 
the contribution of intrinsic entropy. 



9.4-6. Entropy production and FTs The total entropy production involved in one 
forward transition p at time r can be derived from the general expression (126) and 
by using the ratio of the rates (245) as 

As\°\t) = hi ^ ^ = Asp + ASp + qp/T. (246) 

It consists of three contributions. The first is the change in stochastic entropy, 

Asp(T) = -lnb„+(r)/p„-(r)]. (247) 

The second denotes the change in the intrinsic entropy (230) of the system which 
consists here of enzyme and surrounding solution. The third term arises from the 
dissipated heat (233,235). 

Summing over all reactions taking place up to time t and adding the concomitant 
change in stochastic entropy, As = — lnp„j (i)+lnp„Q (0), one obtains the total entropy 
production along a trajectory 

As'°' = A,s + ^ a,[ASp^ (r,) + qp^ (r,)/T], (248) 
j 

where Uj = ±1 denotes the direction in which the transition pj takes place at time r, . 

The arguably most relevant situation for an enzyme modelled by discrete states 
is a NESS generated by non-equilibrated solute concentrations and/or an applied 
external force in the case of a motor protein. In such a NESS, one has the SSFT (65) 
for the total entropy production as defined in (248). 
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9.4-7. Time- dependent rates and work So far, it was implicitly assumed that the 
rates are time-independent. Time-dependent rates could arise either since the 
concentrations of the solutes are externally modulated (or, in a finite system, depleted 
due to the action of the enzymes) or since the forces applied to motor proteins are 
time-dependent. The ratio of the rates is then still constrained by (243, 244). However, 
under such time-dependent external conditions characterized by a parameter A(t), the 
thermodynamic state variables En , Sn and Fn can become time-dependent as well. In 
consequence, there are contributions to the first law and to entropy production even 
if the enzyme remains in the same state. Specifically, if the enzyme remains in state 
n, in analogy to (212) the first law becomes 

dwn = dxF^dX - {dxE„ - TdxSn)dX = dxEndX +dq^. (249) 

Hence, there is exchanged heat, dqn = —TdSn, even if the system remains in the same 
state whenever the intrinsic entropy depends on a changing external parameter (like, 
e.g., the concentration of the solutes). 

These expressions of heat and work resemble those of quasistatic processes as they 
should since it is implicitly assumed that the distribution of microstates that contribute 
to the state n adapts (almost) instantaneously to thermal equilibrium. Consequently, 
these contributions to work and heat do not enter the FTs non-trivially. 

9.4.8. Experiments: Fl-ATPase Apart from free energy reconstructions described 
above, experimental work using the concepts of stochastic thermodynamics is still 
scarce. For the Fl-ATPase, two groups have published work pointing in this direction. 
In an intriguing example of exploiting the SSFT, the torque exterted by the Fl-ATPase 
on a bead in an optical trap could be measured without knowing the friction constant 
of the bead [481]. The implicite assumption, however, with this type of analysis is that 
no further dissipative mechanisms exist. In another study of this molecule [482, 483], it 
was inferred that this motor transfers almost the full free energy from ATP hydrolysis 
into loading the elastic element connecting the motor with the bead. 

9.4.9. Biochemical reaction networks The formalism described above for a single 
enzyme can easily be extended to networks involving several types of (different) 
enzymes [84] or ordinary chemical reactions networks using chemical master equations 
[228, 419]. Specific examples for which the distribution of entropy production has been 
calculated arc [484, 485, 486]. 

10. Autonomous isothermal machines 

10.1. General aspects 

Enzymes and molecular motors as described in Sect. 9 from the stochastic 
thermodynamics perspective provide a paradigm for isothermal machines. In contrast 
to heat engines, which in their classical form are the archetypical thermodynamic 
machines and which in their stochastic version will be described in Sect. 11, isothermal 
machines do not transform heat but rather chemical energy into mechanical work (or 
vice versa) while the temperature of the surrounding medium remains constant. 

A general classification relevant to machines is whether or not they operate 
autonomously. In the stochastic setting, an autonomous machine will typically 
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correspond to a NESS driven by externally imposed time-independent boundary 
conditions. Any molecular motor is a typical example of such an autonomous 
isothermal machine, since single molecule assays typically provide conditions of 
constant non-equilibrium concentrations of "fuel" molecules like ATP. For a non- 
autonomous machine, some time-dependent external control is required that "leads" 
the machine through its cycle. Building reliable artificial molecular motors in the lab 
still constitutes a major challenge as reviewed in [487, 488, 489, 490, 491, 492]. 

In this section, we present a systematic theory for isothermal autonomous 
machines in the discrete state version based on the representation in terms of cycles of 
the underlying network. In principle, this approach includes earlier models based on 
continuous coordinates diffusing in a ratchet potential that depends on the chemical 
state of the motor since any continuum model can be discretized. The emphasis 
here, however, is on the basic principles, and, in particular, how the thermodynamic 
constraints are implemented consistently. 

An important quantity for any type of machine is its efficiency defined as the 
ratio between the power delivered by the machine and the rate of "fuel" consumption. 
Thermodynamics constrains this efficiency by 1 for isothermal machines and by the 
Carnot efficiency for thermal heat engines operating between heat baths of different 
temperature. In both cases working at the highest possible efficiency comes at the cost 
of zero delivered power since reaching the thermodynamic bound requires a quasi- 
static, i.e., infinitely slow operation. A practically more relevant question then is 
about efficiency at maximum power (EMP). We will see that in this thermodynamic 
framework rather general expressions for power, efficiency and efficiency at maximum 
power emerge. 

It would be interesting to pursue these issues also for periodically driven machines, 
which are one step more complex than the autonomous ones. While there is a vast 
literature on how to generate transport by periodic modulation of system parameters 
as reviewed in [415, 493, 494, 495] the problem of efficiency and efficiency at maximum 
power, however, seems not to have been addressed systematically for such stochastic 
machines. One reason is that even making explicit statements about the periodic 
steady state is much harder than for the NESS engine at constant external parameters. 

10.2. General framework for autonomous machines: Cycle representation and 
entropy production 

An autonomously operating device or machine can be modelled as a Markov process 
on a network in a steady state. Transitions between different states in this NESS 
depend on rates that reflect both the coupling of the machine to reservoirs with 
different chemical potentials for solutes or particles and external forces or loads. These 
non-equilibrium conditions can be expressed by generalized thermodynamic forces or 
"affinities" Fk as listed in Tab. 1. 

For a systematic presentation it is useful first to recall the representation of a 
NESS in terms of cycle currents [189, 190], see Fig. 3. Rather than summing over 
the individual transitions [mn) or reactions p as we have done so far, in a NESS 
probability currents can be expressed by a sum over directed cycle currents 

3a = ji - ]- = J + - .?a /.?a+) = Ja+(1 " \{ ^^pl^t) (250) 

where a labels the cycles and and j~ denote the inverse mean times required for 
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Table 1. Affinities and generalized distance for isothermal machines 



process 


affinity 


gen. distance dJ' 


linear motion 


force f/T 


linear distance d 


rotation 


torque N/T 


angle A.<f> 


particle transport 




(typically) 1 



chemical reaction —Afi/T (typically) 1 



completing the cyele in forward and backward direction, respectively.! These forward 
and backward (probability) currents can be expressed in a diagrammatic way by the 
transition rates of the whole network (not just those of the respective cycle). However, 
the ratio ja/jti is given by the ratio between the product of all backward rates and 
the product of all forward rates contributing to the cycle a. 

Thermodynamic consistency as formulated in (139) or (245) allows to express this 

ratio 

\{w-/w+ ^cM-^Sa) (251) 

by the sum 

^Sa = ilp/T + A5p) = qa/T + ^ A5p = Qa/T (252) 

of the entropy changes in the reservoirs and heat baths associated with this cycle. The 
last equality recalls the definition of heat under strict steady state conditions which 
includes the quasi-static refilling of the reservoirs as discussed in Sect. 9.4.3. With 
the first law (239) summed along a cycle, this entropy change can also be written as 

ASa = {wr''^ + "") /T. (253) 

This representation alluding to the definition of a "chemical work" introduced in Sect. 
9.4.3 becomes convenient when discussing the efficiency. 

Alternatively, expressed in terms of affinities, the entropy change associated with 
a cycle becomes 

ASa=Y,daJ'k, (254) 

k 

where is a generalized distance conjugate to the force Tk as listed in Tab. 1. To 
each affinity J-k-, there corresponds a conjugate fiux or current 

Jk ^ 5](j+ - J-)d^a = E-?«^[l - cxp(-A5,)]d^ (255) 

a a 

describing the rate with which the respective quantity is "processed" by the machine. 
The mean entropy production rate can be written as 

a ^ Y.(ja - ^a^^^- (256) 



j+[l-exp(-A5,)]A5a = ^ JfeJ-fc. (257) 



a k 



These expressions are exact and do not imply any linear response assumption as the 
final bilinear form may suggest. 

f The directed cycle current ja = {Ja) is the mean of the fluctuating current J a introduced in Sect. 
6.4. 
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10.3. Power and ejficiency 

10.3.1. Input and output power A device or machine is supposed to deliver some 
output from consuming some input. Characteristically for nano-machines, the role of 
output and input can easily be reversed as it depends on the signs of the corresponding 
affinities. Input has to be offered to the machine with a positive affinity Ti > Q whereas 
output is associated with a current or flux that is opposite to an applied affinity J"o < 0. 

Quite generally, for an isothermal machine in a NESS, the total rate of production 
of output and input, Pq and P^, respectively, is given by the product between a pair 
of corresponding flux and affinity according to 

Po,i = ^osJos{TJ-o,i) (258) 

= eo,. ^ J + - eM-^Sa)]dT{TFo.,) (259) 

a 

where Eq ^ — 1 and e.i = 1 reflect the fact that the output is delivered against an 
external load J-'o < 0. Expressed in terms of a cycle-specific work input 

W,,a = F^dl (260) 

and work output 

Wo^a = -J^od:, (261) 
the power can also be written as [496] 

Po.i = ^ja['^ ~ CXp{w,^a ~ Wo,a)]TW{o,i}a- (262) 
a 

In the contribution of each cycle, the expressions (259, 262) separate a system 
specific kinetic prefactor, j+ = jaii^k}), from the remaining thermodynamic 
quantities. 



10.3.2. Efficiency and efficency at maximum power (EMP) The efficiency of a 
machine is defined as the ratio 

V = Po/P.- (263) 

It has occasionally been argued that the traditional definition of efficiency (263) should 
be modified for molecular motors pulling cargo in order to include the "work" required 
for overcoming Stokes friction even in the absence of an external force [497, 498]. 
More recently, a "sustainable" efficiency has been suggested as an alternative concept 
[499, 500]. In this review, we keep the traditional expression (263). 

For the paradigmatic case of just two non-zero affinities, the entropy production 
rate (257) becomes 

a = (P, - Po)/T > (264) 

implying that efficiency of isothermal machines is bounded by < < 1. Working at 
the highest possible efficiency comes at the cost of zero delivered power since reaching 
the thermodynamic bound requires a quasi-static, i.e., infinitely slow operation. A 
practically more relevant question then is about efficiency at maximum power. 

The notion of efficiency at maximum power (EMP) requires one or several 
parameters {A^} with respect to the variation of which Po can become maximal, i.e., 
P* = max{A.} Po = Po({X*})- EMP is then given by 



ij* EE p;/p,({a:}). 



(265) 
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In general, the result for EMP will depend strongly both on the choice and the allowed 
range of the variational parameters {A^} [496, 501] which is a fact occasionally ignored 
when statements about the EMP are made. In particular, one should distinguish 
variation with respect to the externally imposed affinities from those with respect to 
structural or intrinsic parameters of the machine. Examples for the latter are the 
topology of the network and common prefactors for forward and backward rates that 
leave their ratio and thus the thermodynamics invariant. 



10.4^. Linear response: Relation to phenomenological irreversible thermodynamics 

At this point, it is instructive to consider a machine operating close to equilibrium and 
to cast the results into the framework of linear irreversible thermodynamics [502, 503]. 
This theory truncates an expansion of the fluxes in the first order of the affinities, i.e., 
assumes that 

with the Onsager coefficients Lki- By expanding (255) for small affinities and using 
(254), we obtain for the Onsager coefficients the cycle representation 

Lki=Y.jr''d':dl (267) 

a 

Here, j^'^'^ = jt{{-f'k} = 0) is the equilibrium forward current of a cycle a. In this 
approach, the Onsager symmetry Lki — Lik is satisfied automatically. Similarly, the 
rate of total entropy production (257) becomes 

a « ^j + (A^,)2 ^Y.LkiM. (268) 

a kl 

In this lowest order, power input and output (259) become 

Po,i — ^o,iJo,i{TTo,i) — eo>iT'Y^Li^o,i}k^kJ'o,i- (269) 

fc 

In the paradigmatic case of two affinities, for fixed input affinity J-^ > and 
choosing the output affinity as variational parameter To , maximum power is reached 
for 

T: = -L„Tj2Loo (270) 

leading to an EMP of [504] 

?7* = LlJ[4LooLu - < 1/2. (271) 

The upper bound imposed by the positivity of entropy production is realized for a 
degenerate matrix of Onsager coefficients, 

LooLii = LoiLio, (272) 

which implies that Jo ~ Ji- Possible realizations of this structural condition are (i) 
all unicyclic machines and (ii) tightly coupled multicyclic machines. These two classes 
and the third one of weakly coupled multicyclic machines will be defined and discussed 
in the next sections. 
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10.5. Unicyclic machines 

Unicyclic machines consist of only one cycle which ahows to drop the cycle index a in 
this section, see Fig. 3 for an example. In general, the power delivered and used by a 
unicyclic motor becomes with (262) 

Po,,^Tj+[l-e^"-^^]wo.,. (273) 

Its efficiency rj = Wo/wi depends thus trivially only on the externally imposed affinities 
and the intrinsic properties d°'^ but is independent of the detailed kinetics. In the 
regime < Wo < Wi, the motor will work as intended. For Wo = Wi, the motor has 
optimal efficiency rj ~ 1 but does not deliver any power since it then cycles as often 
in forward as in backward direction. 

The concept of EMP requires to identify the admissible variational parameters. 
A simple and physically transparent choice is to fix the input Wi and vary the output 
Wo, e.g., by changing the applied force or torque in the case of a molecular motor. 
The condition dPo/dwo — leads to the implicit relation [496] 

w^=w:+\n[l + w*J{l + Xow:)] (274) 

for the optimal output w* at fixed input Wi with 

Xo = d\nj+/dwo w x°J-^ + 0{w„Wo). (275) 

These expressions can easily be evaluated for any unicyclic machine with specified 
rates which determine the non- universal 

The linear response regime is defined by the condition Wo < Wi \. By 
expanding (274), one obtains 

jf ^ w*Jw, « 1/2 + + l/2)u;,/8 + 0{w^) (276) 

which shows how system specific features like the coefficient a;"'^ enter EMP beyond 
the universal value 1/2. This expression proves that, depending on the value of the 
non-universal parameter x°^, EMP may well rise beyond the linear response regime 
as found first in a case study of molecular motors [505] . These results seem to be at 
variance with another study along similar lines where a universal bound of 1/2 was 
found for EMP [499]. The difference, however, is that the latter authors constrain the 
optimization to a parameter space that leaves the stationary distribution invariant 
which seems to be a somewhat artificial condition. Further analytical and numerical 
results for EMP of unicyclic machines using Wo or both, Wi and Wo, as variation 
parameters can be found in [496] . Bounds on EMP on simple unicyclic machines have 
been derived in [506]. 

10.6. Multicyclic machines: Strong vs weak coupling 

A discussion of multicylic machines along similar lines does not require much additional 
conceptual effort [496] . The crucial distinction becomes the one between "strong" (or 
tight) and "weak" (or loose) coupling first introduced within the phenomenological 
linear response treatment in [504] and later stressed by van den Brocck and co-workers 
mostly in the context of heat engines as reviewed in the next section. In a strongly 
coupled multicylic machine, any cycle containing the input transition also contains 
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the output transition (assuming for simplicity that input and output affect only one 
transition each). For such strongly coupled machines exactly the same formalism as 
for unicyclic machines applies with the only caveat that j+ appearing there is now 
given by j+ = j+ where the sum runs over all cycles that include input and 
output transition and the j+ are the corresponding forward cycle currents. Thus, 
such strongly coupled machines obey the same relations for efficiency and EMP as 
unicyclic machines. 

In the weak coupling case, there are cycles containing the input but not the 
output transition. Running through such a cycle the machine "burns" the input 
without delivering any output which clearly decreases the efficiency. In particular, it 
turns out that in the linear response regime, EMP is less that 1/2, but may still rise 
when moving deeper into the non-equilibrium regime [496]. 

10.7. Efficiency and EMP of molecular motors 

One important class of potential applications of the theory just described are molecular 
motors that transform chemical energy into mechanical energy (or vice versa). As 
mentioned in the introductory section above, two general types of models must be 
distinguished for these systems. 

For dynamics on a discrete set of states, efficiency (rather than EMP) has been 
investigated recently for various models [507, 508]. Genuine EMP has been studied 
for both the simplest unicyclic and a simple multicycle network in [505]. It would be 
interesting to do so for more intricate models like, e.g., the one introduced in [405], 
but also for artificial swimmers like the one discussed in [509] . 

Traditionally, efficiency of molecular motors has been studied within ratchet 
models where the motor undergoes a continuous motion in a periodic potential that 
depends on the current chemical state of the motor [510, 511, 413, 414, 415, 416]. 
Dissipation then involves both, the continuous degree of freedom which should be 
treated along the lines discussed in Sect. 2 and the discrete switching of the potential 
due to an enzymatic event. Model systems of this type have been investigated in 
[403, 512, 513, 514, 515]. For a recent study of EMP in such a continuum description, 
see [516]. 

There is a second motivation for studying such models combining discrete with 
continuous dynamics. In the typical experimental set-up for measuring the efficiency of 
a molecular motor under load, an external force or torque is applied to a micron-sized 
bead that is connected to the molecular motor like in the recent example of the rotary 
motor Fl-ATPase [482, 483, 517]. The discrete nano-sized steps of the motor become 
visible only through monitoring the biased Brownian motion of the bead which clearly 
is continuous. For a comprehensive description, both dissipation in the discrete steps 
of the motor and the one associated with the continuous motion of the bead should 
be combined [518]. 

11. Efficiency of stochastic heat engines 

11.1. Carnot, Curzon-Ahlhorn and beyond 

In classical thermodynamics, a heat engine, delivering work ~W by extracting heat 
—Qi from a hot bath at temperature Ti and releasing heat Q2 into a cold bath at 
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temperature T2, has an efficiency 



V=\W\l\Qi\<fic = l-T2lTi 



(277) 



limited by the Carnot efficiency rjc which provides a universal bound that follows 
from combining the first with the second law. Reaching the upper bound comes at 
the price of zero power since this condition requires a quasistatic, i.e., infinitely slow 
operation. A practially more relevant efhciency is the one at maximum power, 77*, 
which becomes well-defined only if the parameter space available for the maximization 
is specified. For macroscopic thermodynamics, introduction of this problem is often 
attributed to Curzon and Ahlborn [519] even though their result has been described 
earlier, see [520] and the comment made in Ref.l of [501]. Subsequent work for 
macroscopic engines pursued under the label of finite-time thermodynamics is reviewed 
in [521, 522, 523, 524]. 

Curzon and Ahlborn (CA) assume ordinary heat conduction between the baths 
and the engine that is supposed to operate without further internal losses. By 
optimizing the power with respect to (i) the temperature difference responsible for 
the heat exchange between the baths and the machine, and (ii) the duration of the 
two isothermal steps (while fixing a constant ratio between the time allocated to 
isothermal and adiabatic steps) they obtained for EMP the expression 



which is independent of the thermal conductivities between baths and engine. 

Whether or not the CA result can claim more universality that under the original 
"endoreversible" assumptions, or is even a bound on EMP, is a subtle, if not even 
ill-defined, issue since maximum power depends crucially on the admissible parameter 
space. Beyond the original assumption there are conditions like for a cascade of 
intermediate engines [525, 526] and for "weak symmetric dissipation" [527] where 
CA can be shown to hold for a rather reasonable choice of variational conditions. 
Numerical simulations of finite-time Carnot cycles for a weakly interacting gas have 
been analyzed for efficiency and EMP in [528, 529, 530]. 

A related question is the range of universal validity of the expansion in (278) for 
EMP. For tightly coupled machines defined by an output work fiux that is proportional 
to the heat flux taken from the hot reservoir, the leading term, ?7c/2, follows from 
simple linear irreversible thermodynamics for fixed input and variable output [504]. 
Such a result will hold both for macroscopic as for small engines. 

The question of efficiency and EMP are indeed as relevant and applicable to small 
engines or devices as they are for macroscopic ones. The new aspect concerns the role 
of fiuctuations not in the sense that a fiuctuating efficiency is defined which might lead 
to ill-defined results given the fact that sometimes the heat taken from the hot bath 
would be zero or even negativ. One rather keeps the definition (277) but now W and 
Q are mean values that are determined by averaging over the fiuctuations. J A main 
advantage of a stochastic approach compared to the macroscopic phenomenological 
one is the fact that a thermodynamically consistent kinetics valid beyond the linear 
response regime can easily be imposed. 

Whether the CA result has any relevance to these small thermal engines has been 
one of the main issues in the field especially since the coefficient 1 /8 in the second term 

X The inequality (277) is a trivial consequence of the IFT for total entropy production, if the latter 
quantities are expressed by fluctuating work and heat contributions [531]. 



= 1 - (T2/Ti)i/2 « r;c/2 + ifd^ + 0(r?S)- 



(278) 
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of an expansion of efficiency at maximum power in i]c was found in quite different 
systems [261, 532]. For autonomous machine, i.e., in the steady state regime, the 1/8 
is indeed universal if the system possesses an additional (left-right) symmetry [533]. 
It should be stressed, however, that getting this coefficient requires a second (intrinsic 
or structural) variational parameter beyond the output control required for getting 
the 1/2. For such steady state machines, beyond the second term in the expansion 
(278), the fuU CA result is irrelevant. 

On the other hand, for small cyclic machines, which can be treated formally in a 
spirit closer to OA's original approach, obtaining the factor 1/8 requires a symmetry 
in the exchange with the hot and cold bath [533]. Moreover, for such a machine it is 
possible to obtain the CA result over the full temperature range for certain conditions 
[261, 533]. 

In the following, we decribe paradigmatically how small heat engines or devices 
fit into the stochastic framework from which these and further results for both 
autonomous (steady state) and periodically driven machines can easily be derived. 
We focus on both the formal similarities and differences with the isothermal machines 
and the issue of efficiency of maximum power. 5 Even though we restrict the following 
discussion to heat engines, similar concepts can be applied to refrigerators, see, e.g., 
[535]. 

11.2. Autonomous heat engines 

For understanding both the general issues and the necessary ramifications of the 
comparably simpler framework introduced in Sect. 10 for the isothermal case, it 
is helpful to have a few specific examples in mind. 

11.2.1. Biittiker-Landauer (BL) and Feynman ratchet Transport of a colloidal 
particle in a periodic potential can be induced by an external force at constant 
temperature as discussed in Sect. 2. As an alternative, in the absence of an external 
force, a spatially periodic temperature profile (out of phase with the potential) will 
also lead to net motion as discussed by Biittiker [536], van Kampen [537] and Landauer 
[538]. This set-up is one example of noise- induced transport which is comprehensively 
reviewed by Reimann [415]. From a more thermodynamic perspective, and in the 
presence of an additional opposing external force, such a Biittiker-Landauer (BL) 
ratchet is a simple example for a stochastic heat engine that transforms heat into 
mechanical work. One can then ask for the efficiency of such a device which is 
a subtle question especially when using the overdamped limit for a discontinuous 
temperature profile [14, 539, 540, 541, 542, 543]. The optimization of such a device 
for maximal power has been studied both for variation of the external force [544, 545] 
and for variation of the intrinsic potential [546, 547] . These issues become technically 
simpler in discretized versions [532, 548, 549, 550, 551] as in the simple model sketched 
and described in Fig. 7. This system can also be seen as a simplified (one degree 
of freedom) version of the famous Feyman ratchet [552] that as a paradigm for 
rectification of thermal noise has its own conceptual subtleties [553, 14]. The Feynman 
ratchet has inspired various model systems which have been analyzed both analytically 
and in numerical simulations [554, 555]. 

S Universality of the cfBcicncy if other quantities are optimized has been studied in [534] . 
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Figure 7. In a BL ratchet, a particle preferentially climbs a potential barrier with 
height E over a distance d\ while in contact with a heat bath at T\ . It slides down 
a distance d2 on the cold side with T2 = T\ — AT. This temperature-difference 
driven motion to the right persists for a small enough force / < pulling to the 
left. 




Figure 8. In a thermoelectric device, simplified here as a quantum dot with 
a single relevant energy level E, electrons are transported on average from a 
hot reservoir with /ii and T\ to a cold reservoir with ii2 = Hi + A/< > /ii and 
T2 = T\ — AT using heat from the hot reservoir. 



From an experimental perspective, realizing such a ratchet in aqueous solution is 
not straightforward since one needs significant temperature differences on rather small 
length scales as realized in single particle studies of thermophoresis, see, e.g., [556]. 

11.2.2. Electronic devices In electronic devices, temperature differences can more 
easily be imposed as it is done, e.g., in thermoelectrics where temperature differences 
are exploited to transport electrons against an electro-chemical potential. For such 
systems, thermodynamic considerations have been emphasized by Linke and co- 
workers who pointed out that such machines can indeed operate at the Carnot limit 
[557, 558]. More recent studies based on simple models for quantum dots have 
addressed in particular the issue of efficiency at maximum power [559, 560, 561]. 
The simple paradigm discussed in [559] is sketched in Fig. 8. 

A particularly intriguing aspect of such devices is the observation that in the 
presence of a magnetic field the Onsager-Casimir symmetry relations, in principle, 
seem to allow Carnot efficiency at finite power [562] . This issue deserves further study 
through the analysis of microscopic models as the one suggested in [563]. 

Likewise, any photo-electric device is also coupled to a reservoir of rather high 
temperature since the photons being absorbed from the sun come with the black-body 
distribution of the sun's temperature. Therefore, photo-electric devices are amenable 
to a thermodynamic description focussing of efficiency and EMP, see, e.g., [564]. 
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Figure 9. Common throe-state diagram for the simphfied BL ratchet (Fig. 7) 
and the thermo-electric device (Fig. 8). For the BL-ratchet L and R refer to the 
particle sitting in the mininum and E corresponds to the particle being on the 
barrier top. For the electronic device, L and R refer to the electron being in the 
left or right reservoir. E corresponds to the electron sitting on the quantum dot. 
In both cases, the state R should be identified with L after the electron or particle 
has been transported from left to right thus completing the cycle. The log-ratio 
of the transition rates is given in Tab. 2. 



11.2.3. General theory For any discrete autonomous heat engines in contact with 
heat baths of at least two different temperatures, it must be specified for each 
transition at which temperature it takes place, i.e., with which heat bath the machine 
is in contact at this particular transition. As in the isothermal case, the assumption 
of local detailed balance implies thermodynamic constraints on the ratios of forward 
and backward rates as given in Fig. 9 and Tab. 2 for the two examples introduced 
above. 

The thermodynamic constraints imply that the total entropy production along 
a cycle still fulfills (251). For the representation (254), one needs the affinities with 
the corresponding conjugate distances entering the conjugate fiuxes given for the two 
examples in Tab. 3. The general differences compared to the isothermal case arise 
from the presence of (at least) two different temperatures. First, there is a new 
affinity associated with the two heat baths with energy fiow as the corresponding 



Table 2. Ratio of rates for the devices showrn in Figs. 7 and 8 with their network 
representation Fig. 9. 



BL ratchet -{E -f |/|rfi)/Ti {E - \ f\d2)/T2 

thermo-electric device (/xi — E)/Ti [E — fj,2)/T2 



Table 3. Characteristic quantities for the two example of unicyclic heat engines 
shown in Figs. 7 and 8. 



relevant affinities 



in 

linear response 



conjugate 
distance d!' 



Input 

.(1) ^ T, 



2Wi 



Output 

ToWn 



BL 
ratchet 



1/T2 - l/Ti 
f/TiJ/T2 



AT/T^ 

f/T2..f/T2 



E 
di,d2 



E+\f\di 



AT/ff 
-{Afl/T2 + ^llAT/Ti) 



\f\idi+d2) 



thermoelectric 
device 



I/T2 - l/Ti 

^J■l/Tl - /i2/T2 



E 
1 



E-fii 



^J■2 - Ml 
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flux. Second, if matter is transported between batlis of different chemical potential 
and different temperature, the corresponding affinity involves the two temperatures. 
As a consequence, in linear response, the latter affinity carries both a A/x and a 
AT term. Finally, a force applied to a particle in a thermal ratchet subject to two 
different temperature requires to introduce even two affinities with this force. As in 
the isothermal case, for an autonomous heat engine the total entropy production rate 
can be expressed by affinities and conjugate fluxes according to (257). 
On the cycle level, the total entropy change becomes 

A5a = giiVT^i+9fVr2, (279) 

where we use the heat as appropriate under NESS conditions which fulfills first laws 
of the type 

^mech(1^2) ^ ^ehcm(1.2) ^ _(1,2) ^ ^^(1,2) (280) 

where I^e'iP = —AE^"^ is the change in internal energy of the system arising from the 
transitions associated with the respective heat baths labelled by superscripts. With 
this relation, the total entropy change along a cycle (279) can also be expressed by 
the heat extracted from the hot reservoir as 

A^a = ~qi'^Vc/T2 + (W^r^l^ + <^™)/T2 = W.^aVC ~ Wo^a, (281) 

where the work terms refer to the sum of the contributions from the respective bath 
contacts. The definition of dimensionless input Wi_a is motivated by the fact that for a 
heat engine the input is the heat extracted from the hotter bath. Dimensionless output 
denoted by Wo_a is either mechanical or chemical work delivered by the machine. 

The power of the machine can now be expressed analogously to the isothermal 
case as 

PoA = ^o,i^ja[^ - exp(-«7i,a??C + Wo,a)]T2W{od}a- (282) 
a 

where the occurence of the Carnot efficiency rjc in the exponent compared to the 
isothermal case (262) is crucial. In the affinity representation, the difference to the 
isothermal case is even more drastic since output and input power can no longer be 
written as simple products of a pair of conjugate flux Jo^i and affinity T!Fo,i as in 
(259). One could have anticipated this complication since with two baths it is not 
obvious which temperature should be chosen in (259) when trying to generalize to the 
non-isothermal case. 

11.2.4- EMP for unicycUc machines For unicyclic machines (and hence dropping the 
cycle index a), maximizing the power Pg with respect to the output Wo, which would 
physically correspond to varying the external force or chemical potentials, leads to the 
analogue of (274) in the form 

- {w; + ln[l + w*J{l + Xow:)])/vc. (283) 

This relation implies immediately the universal if = w*/wi « i]c/2 + 0{tj^) in the 
linear response regime which can also easily be obtained from a phenomenological 
treatment analogously to the one presented in Sect. 10.4 [525]. 

Maximizing the power with respect to the input variable Wi leads to 



(284) 
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with 

X, = d\nj+/dw, « x^'^ + Oi7^c,wo). (285) 

As the respective column in Tab. 3 show, Wi involves an intrinsic parameter of the 
machine like the relevant energy level. Combining the relations (283,284) and varying 
both Wo and Wi leads to the EMP 

V** = Vc/[l - {xo + xjric) ln(l - vc/x^)] (286) 
« rjc/2 - \{2xl^ + l)/.Tn'?^/8 + 0{nl) (287) 

which shows that the second order coefficient is system-specific. 

It can be checked that for a unicyclic device with spatial symmetry, for which the 
current j reverses sign when the affinities J-^ change sign, the square-bracket prefactor 
of the second term is indeed -1 thus recovering the overall 1/8 as previously derived 
by extending the phcnomonological irreversible thermodynamics approach to second 
order [533]. 

For an explicit evaluation of the EMP (286) one needs the specific form of 
Xo,i = Xo^i^WojWi) which requires assumptions on the specific rates beyond the 
constraints imposed by thermodynamics exploited so far. For the mechanical BL 
ratchet, it is interesting to note that even for di ^ d2, an explicit calculation for 
W2 = Wi = 1 (and the other rates as given in Tab. 2) recovers the coefficient 
1/8 despite the obvious breaking of the left- right symmetry. The case (i2 = is 
discussed for the full temperature range in [532], where, not surprisingly, deviations 
from the CA result are found. The thermoelectric device is treated in [533, 559]. For 
a photo-electric device, explicit results can be found in [564], where also the role of 
non-radiative transition is discussed. Further examples of EMP in a three and five 
state network have been discussed in [501]. 

11.3. Periodically driven heat engines 

The autonomous machines just discussed reach a NESS since they arc permanently 
connected to both heat baths. For a periodically driven heat engine, contact with 
either one bath or, in an adiabatic step, with none, is periodically enforced externally. 

11.3.1. Brownian heat engine Within stochastic thermodynamics such a model was 
introduced in [261] for a Brownian particle in a time-dependent potential, see Fig. 
10. Optimizing for both, the potential and the time-interval spent in the isothermal 
transitions, EMP for fixed Ti,2 was shown to be 

r^* = 7?c/(2 - aijc) ^ Vc + {a/^Wc + 0{ifc) (288) 

where 

a^l/\l + {^l{T,)/^x{T2)f'^] (289) 

is a system-specific coefficient given by the temperature-dependence of the mobility 
niT). If the latter is independent of temperature, one recovers the coefficient 1/8. 
Since < a < 1, the expression (288) implies the bounds 

??c/2 < 7f < vc/{2 - vc) (290) 

on EMP later also derived under the assumption of "weak" dissipation which leads to 
a quite similar formalism [527, 565]. Further ramifications and classifications of such 
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Figure 10. Paradigm for a stochastic heat engine based on a colloidal particle 
in a timc-dcpcndent harmonic laser trap in consecutive contact with a hot (Tfj) 
and cold (Tc) bath [261]. The steps 1 and 3 are isothermal; the steps 2 and 
4 are instantanous and adiabatic during which the distributions are pi, and pa, 
respectively. 

bounds have been discussed by Wang and Tu [566, 567] and in [568]. The Onsager 
coefficients for a Hnear response treatment of this heat engine have been defined and 
calculated in [569]. 

A micron-sized heat engine based on the Stirling version of the scheme shown in 
Fig. 10 has just been realized experimentally [570]. The colloidal particle as "working 
fluid" in a laser trap acting as the analogue of a piston can be heated locally thereby 
realizing the contact with a hot bath. By varying the cycle time both a maximum in 
the power at finite time and the approach to the maximal efficiency in the quasi-static 
limit could be demonstrated. For further interesting comments on this experiment, 
see [571]. 

11.3.2. Quantum dot A similar analysis can be applied to a finite-time Carnot cycle 
of a quantum dot that can be connected to two different reservoirs similar as the set-up 
shown in Fig. 8 [572]. For a cyclic engine, the energy level E{t) is controlled in both 
the isothermal steps when the dot is connected to either one bath and in the adiabatic 
steps when it is disconnected. Optimizing for the protocol E{t) as well as for the 
duration of isothermal and adiabatic steps, one finds for EMP an expression similar 
to (288). In the limit of weak dissipation, i.e., for small deviations from the respective 
thermal population of the energy level, and symmetric conditions, the coefficient a 
turns out to be a = 'Hca/vc and hence one can here recover the CA result over the 
full temperature range. The distributions of work and heat for such a simple two-state 
engine have been calculated in [573]. 

12. Concluding perspective 

After this long exposition it may be appropriate to recall the basic assumption of this 
approach, to summarize the main achievements and to raise a few open general issues. 

12.1. Summary 

Stochastic thermodynamics applies to systems where a few observable degrees 
of freedom like the positions of colloidal particles or the gross conformations of 
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biomolecules are in non-equilibrium due to the action of possibly time-dependent 
external forces, fields, flow or unbalanced chemical reactants. The unobserved degrees 
of freedom like those making up the aqueous solution, however, are assumed to be fast 
and thus always in the constrained equilibrium imposed by the instantaneous values 
of the observed slow degrees of freedom. Then internal energy, intrinsic entropy and 
free energy are well-defined and, if a microscopic Hamiltonian was given, in principle, 
computable for fixed values of the slow variables. This assumption is sufficient to 
identify a first-law like energy balance along any fluctuating trajectory recording the 
changing state of the slow variables. 

Entropy change along such a trajectory consists of three parts: heat exchanged 
with the bath, intrinsic entropy of the states and stochastic entropy. The latter 
requires in addition an ensemble from which this trajectory is taken. If the same 
trajectory is taken from a different ensemble it leads to a different stochastic entropy. 
Thermodynamic consistency of the Markovian dynamics generating the trajectory 
imposes a local-detailed balance condition constraining either the noise in a Langevin- 
type continuum dynamics or the ratio of transition rates in a discrete dynamics. 

At their core, the fluctuation theorems are mathematical identities derived 
from properties of the weight of stochastic paths under time reversal or 
other transformations. They acquire physical meaning by associating the 
mathematical ingredients with the thermodynamic quantities identified within 
stochastic thermodynamics. The detailed fiuctuation theorems then express a 
symmetry of the distribution function for thermodynamic quantities. An open 
question is whether the probability distributions of work, heat and entropy production 
can be grouped into "universality classes" characterized, e.g., by the asymptotics of 
such distributions, and, if yes, which specific features of a system determine this class. 
The more generally applicable integral theorems often can be expressed as refinements 
of the second law for transitions between certain states. Still, these integral fluctuation 
theorems should not be considered a "proof" of the second law since irreversibility has 
been implemented consistently from the beginning by choosing a stochastic dynamics 
including the local detailed balance condition. 

Conceptually, a major step has been to include feedback schemes based on 
perfect or imperfect measurements into this framework which requires surprisingly 
little additional effort due to the strong formal similarity of stochastic entropy 
with information. Still missing is a full integration of measurement apparatus and 
the erasure process into the thermodynamic balance of the efficiency for specific 
information machines. 

The crucial ingredient for the developments summarized so far is the notion of an 
individual trajectory and the concomitant concept of distributions for thermodynamic 
quantities which represents the main difference compared to classical thermodynamics. 

New insights, however, have emerged from this approach even when focussing on 
averages and correlation functions as we have done in the second part of the review. 
The general fluctuation-dissipation theorem for non-equilibrium steady states shows 
how the response of any observable to a perturbation can be expressed as a sum of 
two correlation functions involving entropy production. In which cases this additive 
relation between response and correlation can be reformulated as a multiplicative one 
using the concept of an effective temperature is still not understood despite some 
insights gained from speciflc case studies. 

Our discussion of molecular motors, machines and devices centered on the notion 
of efficiency and efficiency at maximum power. Despite the fundamental difference of 



CONTENTS 



83 



isothermal engines operating at one temperature as do all cellular ones from genuine 
heat engines like thermoelectric devices involving two baths of different temperature, a 
common framework exists based on the representation of entropy production in terms 
of cycles of the underlying network of states. Clearly, more realistic networks need 
to be studied in the future, in particular, for applications and modelling of specific 
biophysical systems but the basic concepts seem to be identified. One particulary 
intriguing perspective comes from the recent analysis of the energetic cost of sensory 
adaptation using the concepts of stochastic thermodynamics [574]. 

12.2. Beyond a Markovian dynamics: Memory effects and coarse-graining 

The identification of states, of work and of internal energy, i.e., of the ingredients 
entering the first law on the level of trajectories is logically independent of the 
assumption of a Markovian dynamics connecting these states. The crucial step is 
the splitting of all degrees of freedom in slow and fast ones, the latter always being 
in a constraint equilibrium imposed by the instantaneous values of the slow ones. 
Likewise, the identification of entropy production only requires the notion of an 
ensemble which determines stochastic entropy along an individual trajectory. Any 
dynamics guaranteeing that for fixed external parameters compatible with genuine 
equilibrium, this equilibrium will be reached for an arbitrary initial distribution of the 
slow variables could qualify as a thermodynamically consistent one. 

12.2.1. Continuous states A popular choice for a non-Markovian dynamics 
obeying these constraints is Langevin dynamics with a memory kernel that, for 
thermodynamics consistency, determines the correlations of the coloured noise. || 
Under this assumption, the notions of stochastic thermodynamics are well-defined 
and the various fiuctuation theorems hold true as shown quite generally in [343, 578, 
579, 580, 581]. Some of these papers contains illustrations for model systems as do the 
references [582, 583, 584]. One specific motivation to explore such a dynamics arises 
from the recent fascinating experimental data that show how hydrodynamic effects 
lead to a frequency dependent mobility for colloidal motion [585] . 

A somewhat different and more subtle situation occurs if not all slow variables 
are accessible in the experiment or the simulation. The effective dynamics for the 
observable ones will then no longer be Markovian and its specific form in the case 
on non-harmonic interactions between the slow one is typically not accessible. The 
proper identification of, e.g., entropy production is then difficult if not impossible. 
Still, one might be inclined to infer an apparent entropy production by applying the 
rules for Markovian dynamics and to check whether this quantity obeys the FT. In 
a recent study using two coupled driven colloidal particles it turned out that the 
apparent entropy production based on the observation of just one particle shows an 
FT-like symmetry but with a different prefactor for a surprisinglingly large range of 
parameter values. However, there are also clear cases for which not even an effective 
FT can be identified [586] . This type of coarse-graining in the context of the FDT for 
a NESS has been explored in [587]. 

II Stochastic thermodynamics for a non-Markovian dynamics not obeying such a constraint has been 
explored for generalized Langevin equations in [575, 343], for delayed Langevin systems in [576], and 
for Poissonian shot noise in [577]. 
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12.2.2. Discrete states For an underlying dynamics on a discrete set of states 
following a Markovian master equation, one option for coarse-graining is to group 
several states into new "meso-states" or aggregated states. Typically, the dynamics 
between these meso-states is then no longer Markovian. One question is whether 
one can then distinguish genuine equilibrium from a NESS if only the coarse-grained 
trajectory is accessible. For a three state system coarse-grained into a two-state 
system, this issue has been addressed in [588, 589] and, for more general cases, in 
[590, 591]. 

Coarse-graining of a discrete network becomes systematically possible if states 
among which the transitions are much faster are grouped together. From the 
perspective of stochastic thermodynamics, entropy production and fluctuation 
theorems this approach has been followed in [592, 593, 594, 595, 596, 597, 598, 599]. 

12.3. Coupling of non- equilibrium steady states: A zeroth law? 

Besides the first and the second law, classical thermodynamics is founded on a zeroth 
law stating that the notion of temperature and chemical potential for equilibrium 
systems is transitive, i.e., if a system A is in separate equilibrium with system B and 
system C, then upon contact of B and C neither heat nor particle flow will occur 
between these two systems. A natural question is whether a similar equilibration 
also occurs for non-equilibrium systems brought in contact such that they can 
exchange energy or particles. Do then quantities exist resembling temperature or 
chemical potential that govern "equilibration" between such steady states? On a 
phcnomenological level this question has been introduced within the context of steady 
state thermodynamics by Oono and Paniconi [46] and further refined by Sasa and 
Tasaki [47]. For simple one-dimensional model systems like zero-range processes in 
contact a non-equilibrium chemical potential is indeed well-defined [600]. For two- 
dimensional driven lattice gases in contact, numerical work has revealed that in a 
large parameter range such a putative zeroth law and a corresponding thermodynamic 
structure is approximately valid [601, 602]. 

12.4. Final remark 

From its very beginnings, thermodynamics fascinated scientists by posing deep 
conceptual issues that needed to be resolved for understanding and optimizing 
quite practical matters like the design of heat engines. With the experimentally 
realized micron-sized heat engine [570] discussed in one of the last sections of 
this review, these latest developments have brought us back to the very origins 
of classical thermodynamics albeit on quite different time and length scales and, 
quite importantly, with a much refined view on individual trajectories. Indeed, 
without the spectacular advances in experimental techniques concerning tracking 
and manipulation of single particles and molecules, stochastic thermodynamics could 
still have been conceived as a theoretical framework but would have not reached 
the broader appeal that it has gained over the last fifteen years. Whether the 
next decade of research in the field will be dominated by specific applications, most 
likely for biomolecular networks and devices facilitating transport of all sorts, or by 
further conceptual work exploring the ultimate limits of a thermodynamic approach to 
non-equilibrium beyond the Markovian paradigm into feedback-driven, information- 
processing, strongly interacting systems remains to be seen. 
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